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3 This application is a continuation-in-part of U.S. Application Serial Number 09/597,893, 
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5 OBJECTS, which is a continuation-in-part of U.S. Application Serial Number 09/054,419, filed 

6 April 3, 1 998, entitled TARGETED ADVERTISEMENT USING TELEVISION DELIVERY 

7 SYSTEMS, which is a continuation-in-part of U.S. Application Serial Number 08/735,549, filed 

8 October 23, 1996, entitled METHOD AND APPARATUS FOR TARGETED ADVERTISING, 

9 which is a continuation of U.S. Application Serial Number 08/160,280, filed December 2, 1993, 

1 0 entitled NETWORK CONTROLLER FOR CABLE TELEVISION DELIVERY SYSTEM, now 

11 U.S. Patent Number 5,600,364, which was a continuation-in-part of U.S. Application Serial 

12 Number 07/991,074, filed December 9, 1992, entitled REMOTE CONTROL FOR MENU 

13 DRIVEN SUBSCRIBER ACCESS TO TELEVISION PROGRAMMING. 

14 Technical Field 

15 The technical field relates to a method and apparatus for providing interactive virtual 

16 objects that are targeted to subscribers. The method and apparatus specifically relate to 

1 7 monitoring, controlling and managing a content delivery network including an operations center, 

18 a local insertion center, or a subscriber's local terminal for the delivery of interactive virtual 

19 objects and the management of the respective responses to interactive requests. 

20 Background 

21 Television as an advertising medium has undergone significant advances since its 

22 inception in 1939. Modern advances in digital technology now allow viewers to be very 

23 selective in choosing programs to watch. Other advances in digital technology have led to such 

24 programming enhancements as a "tail" on a hockey puck, or an overlaid first down marker on a 

25 football field. The same technological advances allow improvements in the way advertisements 

26 are currently presented. Many sporting events are now presented with virtual object 

27 advertisements included in the video. For example, the broadcast of a major league baseball 

28 game may include one or more virtual object spots in which an advertisement is displayed. The 

29 advertisements are then seen by television viewers, but not by fans who attend the game. 

30 However, the advertisements are fixed, and are not varied according to individual viewers. 
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1 Summary 

2 A system and a method delivers interactive targeted virtual objects to reception sites. A 

3 virtual object is a realistic, synthetic replica of an actual object. The virtual object is viewable 

4 within video programming and may be combined with original video and audio to supplement or 

5 replace portions of the original video and audio content. Virtual objects may be overlaid on 

6 video, partially or entirely obscuring the underlying video. An overlaid object may be static in 

7 nature, such as a graphical icon or the like, or alternatively may be dynamic, such as a video clip, 

8 animation, or scrolling alphanumeric characters, for example. Overlaid objects may be limited 

9 spatially to a fixed portion of the video screen, limited temporally to a given time for display, 

1 0 limited by a combination of both location and time, or tied to a spatially changing portion of the 

1 1 screen that is moving with time. Alternatively, virtual objects may be added to and embedded 

12 within the actual video. Multiple virtual objects may be embedded in the video in a multi-layer 

13 fashion. The virtual object is indistinguishable from the other video content sharing the field of 

14 view. Virtual objects may be interactive in nature. That is, a viewer may select an object and 

15 the selection will initiate a process whereby a reception site takes some action based on the 

16 interactive virtual object or the reception site sends a command to a location designated by the 

17 interactive virtual object to initiate some action. 

1 8 An interactive virtual obj ect management center defines interactive obj ects, and provides 

19 the interactive objects to the operations center for delivery, and the interactive virtual objects 

20 response management guidelines to an interactive obj ect servicing center. An operations center 

21 may process the video signal to allow for the insertion of virtual objects into the video. An 

22 object delivery center serves as a standalone or supplemental system to the operations center to 

23 deliver virtual objects independently of the video with which the virtual objects are to be 

24 associated. A delivery network includes any of a number of different delivery systems to 

25 support the delivery of video and virtual objects from the operations center and the object 

26 delivery center to a local insertion center, or directly to a reception site. The delivery network is 

27 also used to deliver video and virtual obj ects from the local insertion center to the reception site. 

28 The reception site receives the video and virtual objects and associates the appropriate virtual 

29 objects with the video based on targeting algorithms. 
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1 The reception site collects virtual object viewing information and makes the viewing 

2 information available to a local data collection center or a central data collection center using the 

3 delivery network. The local data collection center provides information collected from the 

4 reception site to the local insertion center to assist in the targeting of the virtual objects. The 

5 central data collection center provides information collected from the reception site to the 

6 operations center to assist in the targeting of virtual objects. Alternatively, the reception site may 

7 use the virtual object viewing information and other information stored at the reception site to 

8 locally target the virtual objects at the reception site. The reception site may provide interactive 

9 requests, which are driven by the selection of interactive virtual objects, to an interactive object 

1 0 servicing center using the delivery network. Interactive responses are returned by the interactive 

11 object servicing center to the requesting reception site based on the interactive virtual object 

12 response guidelines provided to the interactive object servicing center by the interactive virtual 

1 3 obj ect management center. 

14 A targeting routine makes use of a viewer's demographic information and viewing habits 

15 to determine those virtual objects that may be most effective when displayed to that particular 

1 6 viewer. In so doing, the targeting routine generates packages of virtual objects targeted to each 

17 viewer, or to groups of viewers. 

1 8 The process of managing the content and the virtual obj ects to be included in the content 

19 begins with a number of configuration and set-up steps. Individual reception site address 

20 information can be collected at the operations center. This information is used to uniquely 

21 identify each reception site and to associate with that identifier necessary information to aid in 

22 the targeting process. The reception site address information may be provided to the operations 

23 center upon installation or activation of the reception site in the viewer's home. Other 

24 information may be collected from various sources, including viewer surveys and marketing 

25 databases correlated by address, zip code+4, Nielsen or Arbitron program rating services, for 

26 example. 

27 Next, reception site groups are determined. This is needed if the management of 

28 information and targeting to individual reception sites is not practical initially, either due to lack 

29 of availability of information to the appropriate level of detail, or lack of technology to control 

30 and deliver virtual objects to an individual reception site. For a number of target categories, 
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1 groups are defined. Examples of target categories include demographic targeting 

2 (age/sex/income) and location, such as Area of Dominant Influence (ADI), for example. Each 

3 target category is then segmented into appropriate groups. For example, the ADI may include 

4 Los Angeles, CA and Washington D.C. New target categories can be added and the groups 

5 redefined after their initial establishment. Anywhere from one to all reception sites may be 

6 assigned to a single group. 

7 For each target category, each reception site is assigned to a group based on the 

8 information collected about the reception site. Once the reception site is assigned to a group, the 

9 group assignment is conveyed to the reception site and stored therein. Alternatively, the 

10 reception site may use information collected locally to assign the reception site to groups. 

1 1 The group assignment information that is stored at the reception site is able to survive 

12 power cycling of the reception site, and other normal service interruptions. Finally, as groups 

13 are modified or group assignments change, reception sites are notified of the changes. 

14 Additionally, the group assignment information may be periodically resent to the reception sites 

1 5 to ensure that newly added reception sites and those that have accidentally lost their information 

16 are up-to-date. 

17 A virtual object location definer system determines where in the content the virtual 

18 objects are to be placed and the rules associated with their placement. Content may be video 

1 9 programming, commercials and advertisements, or electronic program guide (EPG) information, 

20 for example. A virtual object selector system determines those available virtual objects suitable 

21 for placement in a virtual object location. A targeted virtual object management system 

22 determines which reception sites or reception site groups should receive and display which 

23 virtual object for a given virtual object location. The virtual objects and targeting information 

24 are then distributed to reception sites. 

25 After reception sites receive and store the virtual objects and targeting information, the 

26 reception site will place the most appropriate virtual obj ect into virtual obj ect locations based on 

27 the targeting information, and will display the combined content with the overlaid or embedded 

28 virtual object. 

29 The reception site stores information indicating that a virtual object was inserted. The 

30 accumulated history information may be collected from the reception site at a later time for 
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1 review purposes. The unique reception site identification information may also be provided with 

2 the collected data. As mechanisms become available to identify specific viewers in a household, 

3 the system will allow for individual identification information to also be provided with collected 

4 data. Finally, after collection of the reception site viewing history data, the reception site returns 

5 used reception site memory space to the reception site. 

6 A centralized operations center can determine virtual object locations available for virtual 



7 object placement. Alternatively, a local insertion center can determine the virtual object 

8 locations. The operations center can determine the specific virtual objects to be placed in a 

9 virtual object location. Alternatively, the local insertion center may determine the specific 

10 virtual object to be placed in a virtual object location. The reception site itself can determine 

1 1 which virtual obj ect is to be placed in a virtual obj ect location based on its own internal routines. 



1 2 Content, virtual obj ects, and associated targeting/virtual obj ect placement control can be 

13 relayed to reception sites and information extracted from the reception site. The reception site 

14 may reside within a digital cable set top box that has access to a delivery network. Alternately, 

1 5 the reception site may be components of digital television satellite receivers. The reception site 

1 6 may be incorporated into the circuitry of a television, thereby eliminating the need for a separate 

17 control device attached to the television. Alternatively, the reception site may be incorporated 

18 into a personal computer, personal data device, smart phone with a display, or electronic book 

19 device 

20 Description Of The Drawings 

2 1 The detailed description will refer to the following drawings in which like numerals refer 

22 to like items, and in which: 

23 Figure 1 is an overview of the virtual object targeting delivery system; 

24 Figure 2 provides a pictorial representation of virtual objects and virtual object locations; 

25 Figure 3 is an example of an overlaid virtual object; 

26 Figure 4 is an example of an embedded virtual object; 

27 Figure 5 depicts an operations center; 

28 Figure 6 depicts a virtual object definer; 

29 Figure 7 is a pictorial representation of a virtual object location matte; 

30 Figure 8 depicts a virtual object selector; 
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1 Figure 9 depicts a targeted virtual object management system; 

2 Figure 1 0 shows configuration and set-up steps associated with targeting virtual objects; 

3 Figure 1 1 shows a subscriber information database system; 

4 Figure 12 shows a configuration set-up system; 

5 Figure 13 shows a virtual object targeting system; 

6 Figure 14 presents an embodiment of the overall process for assigning targeted virtual 

7 objects; 

8 Figure 15 presents an embodiment of a process used by the virtual object placement 

9 engine to assign virtual objects to virtual object locations; 

1 0 Figure 1 6 presents an alternate embodiment used by the virtual obj ect placement engine 

1 1 to assign virtual objects to virtual object locations; 

1 2 Figure 1 7 presents yet another embodiment used by the virtual obj ect placement engine 

13 to assign virtual objects to virtual object locations; 

14 Figure 18 shows functions of an alternate virtual object targeting routine; 

1 5 Figure 1 9 shows an embodiment of a matrices processing subroutine that is called by the 

16 virtual objects targeting sequence; 

17 Figure 20 shows a subroutine used to select the final groupings of virtual objects to be 

18 sent to the reception sites or group of reception sites; 

19 Figure 21 shows a representation of reception site groupings; 

20 Figure 22 shows an example of a division of available bandwidth; 

21 Figure 23 shows an alternative software program flow for an object targeting routine; 

22 Figure 24 depicts an object delivery center; 

23 Figure 25 presents embodiments associated with the delivery of virtual objects over a 

24 coaxial or fiber cable system to a reception site; 

25 Figure 26 presents embodiments associated with the delivery of virtual objects over a 

26 wireless broadcast system to a reception site; 

27 Figure 27 presents embodiments associated with the delivery of virtual objects over a 

28 satellite broadcast system to a reception site; 

29 Figure 28 presents embodiments associated with the delivery of virtual objects over a 

30 wired data network to a reception site; 
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1 Figure 29 presents embodiments associated with the delivery of virtual obj ects using the 

2 public switched telephony network (PSTN) to a reception site; 

3 Figure 30 presents embodiments associated with the delivery of virtual objects using 

4 wireless personal communications system (PCS) to a reception site; 

5 Figure 31 depicts several embodiments associated with the delivery of virtual objects 

6 using a national or local television broadcaster's signal; 

7 Figure 32 depicts a local insertion center; 

8 Figure 33 depicts an example of a reception site; 

9 Figure 34 depicts a local data collection center; 

10 Figure 35 depicts a central data collection center; 

11 Figure 36 depicts an embodiment of the process performed by the interactive object 

12 process upon receipt of a trigger; 

13 Figure 37 presents an interactive object example; 

14 Figure 38 depicts an interactive virtual object management center; 

15 Figure 39 depicts an interactive object servicing center; and 

16 Figure 40 presents processing performed by an interactive object servicing center. 

17 Detailed Description 

1 8 An overview of the interactive virtual obj ect delivery and targeting system is depicted in 

19 Figure 1 . An operations center 10 performs the processing of a video content signal to allow for 

20 the insertion of virtual objects into the content 36. An object delivery center 15 serves as a 



21 standalone or supplemental system to the operations center 10 to deliver virtual objects 

22 independent of the content with which the virtual objects are to be associated. A delivery 

23 network 1 1 includes any of a number of different delivery systems to support the delivery of the 

24 content 36 and virtual objects from the operations center 1 0 and the object delivery center 1 5 to a 

25 local insertion center 20 or directly to a reception site 30. A delivery network 12 is used to 

26 deliver content and virtual objects from a local insertion center 20 to the reception site 30. The 

27 reception site 30 may be any device or terminal capable of receiving video, including a set top 

28 terminal, a television, a personal computer, a wireless telephone, a wired telephone, a PDA 

29 device, an electronic book, a digital satellite television receiver, or any similar device or 

30 terminal. 
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1 The reception site 30 receives the content 36 and virtual objects and associates the 

2 appropriate virtual objects with the content 36 based on targeting algorithms. The reception site 

3 30 may collect virtual object viewing information and make the viewing information available to 

4 a local data collection center 40 or a central data collection center 50 using a delivery network 

5 13. Alternatively, the reception site 3 0 may retain all virtual obj ect viewing information and use 

6 the information to target virtual objects locally without control from the operations center 10. 

7 The local data collection center 40 provides information collected from the reception site 30 to 

8 the local insertion center 20 to assist in the targeting of virtual objects. The central data 

9 collection center 50 provides information collected from the reception site 30 to the operations 

1 0 center 1 0 to assist in the targeting of virtual obj ects. The interactive virtual obj ect management 

1 1 center 55 provides for the creation and definition of interactive virtual objects. An interactive 

1 2 virtual obj ect, discussed in more detail below, contains virtual obj ect identifying information, the 

13 actual virtual object, an interactive virtual object trigger action 56, and virtual object response 

14 management guidelines 57. An interactive virtual object trigger action 56 defines those actions 

15 which the reception site 30 takes once an interactive virtual object is selected at the reception 

16 site. An interactive virtual object response management guideline 57 may be provided to the 

17 interactive object servicing center 60 by the interactive virtual object management center 55 and 

18 used by the interactive object servicing center 60 to determine the appropriate response upon 

19 receipt of an interactive request from a reception site 30. The reception site 30 provides 

20 interactive requests, which are driven by the selection of interactive virtual objects, to an 

21 interactive object servicing center 60 using a delivery network 14. Interactive responses are 

22 returned by the interactive object servicing center 60 to the requesting reception site 30. 

23 Virtual objects may be realistic, synthetic replicas of actual objects. Virtual objects may 

24 also be caricatures of actual individuals, photographs or other life-like renderings of actual 

25 individuals, cartoon figures, text obj ects, graphical renderings, or icons, for example. The virtual 

26 objects may be animated or fixed. The virtual objects are combined with video and audio to 

27 supplement or replace portions of video and audio in original content 36. As shown in Figure 2, 

28 the reception site 30 may contain or be connected to a display 35 on which the content 36 may be 

29 displayed. An opportunity, advertisement spot, or location, in the content 36 that is available for 

30 the placement of the virtual object will be denoted as a virtual object location 37 henceforward. 
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1 Within the virtual object location 37, one or more individual virtual objects may be assigned, 

2 each denoted as a virtual object 38 henceforward. Multiple virtual object locations, shown as 

3 virtual object locations 37 and 39 may be present in the content 36. Multiple virtual objects, 

4 shown as virtual objects 38 and 40 may be present within the virtual object locations. 

5 As shown in Figure 3, virtual objects may be overlaid on video, partially or entirely 

6 obscuring the underlying video. An overlaid virtual object may be static in nature, like a 

7 graphical icon, as shown by virtual object 42. Alternatively the overlaid virtual object may be 

8 dynamic, like a video clip, animation, or scrolling alphanumeric characters as shown by virtual 

9 object 44. Overlaid virtual objects may be limited spatially to a fixed portion of the video, 

10 limited temporally to a given time for display, or limited by a combination of both location and 

1 1 time. Overlaid virtual objects may also be tied to a spatially changing portion of the video that is 

1 2 moving with time. 

13 Alternatively, as shown in Figure 4, virtual objects may be added to and embedded 

14 within video. In this alternative, the synthetic virtual object 38 could be indistinguishable from 

15 the other video content 36 sharing the field of view as shown by virtual object 46 and virtual 

16 object 48. For instance, todays technology allows for the virtual placement of a billboard at 

17 televised sports events and the placement of a virtual first down marker in televised football 

18 games. 

19 In an embodiment, virtual reality and animation technologies are combined with 

20 advanced digital video techniques to provide realistic interaction of virtual objects within video. 

21 Combining these technologies, a soda can may be synthetically placed in the video, and may 

22 then be made to change over time. This placement and subsequent modification can occur at the 

23 video's source, at an intermediate point within the distribution and delivery path, or at the 

24 reception site 30. Combining the placement of virtual objects with the ability to target specific 

25 virtual objects to specific viewers or groups of viewers allows one household to see a scene with 

26 the soda can for cola, while the next door neighbor sees a root beer soda can, for example. 

27 Virtual objects may be interactive in nature, where a viewer can select a virtual object 35 

28 and this selection will initiate a process whereby the reception site 30 initiates some action or the 

29 reception site 30 sends a command to the location designated by the interactive virtual object 38 

30 to initiate some action. Actions may include linking to a Web site to display content related to 
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1 the interactive virtual obj ect 3 8, initiating a purchase transaction, or initiating a request for more 

2 information about the selected virtual object 38. 

3 The operations center 10 shown in Figure 1 may include a number of systems that act 

4 together in processing the content 36 for the inclusion of virtual objects, for the selection of 

5 appropriate virtual objects to be placed in the content 36, for the targeting of virtual objects to 

6 individual reception sites, and for the packaging and delivery of the content 36 and virtual 

7 objects to reception sites. 

8 Placement of virtual objects can be explicitly selected by the operations center 10, 

9 resulting in the specific selection and placement of virtual objects into content 36. Alternatively, 

10 the placement may be generically defined by the operations center 10. In this alternative, the 

11 reception site 30 performs all the processing associated with selecting the appropriate virtual 

12 object 38 to be placed in the content 36 based on basic guidelines provided by the operations 

13 center 10 and algorithms operating at the reception site 30. 

14 As shown in Figure 5, the operations center 1 0 includes a virtual object location definer 

15 100, a virtual object selector 200, and a targeted virtual object management system (TVOMS) 

16 300. 

17 Figure 6 presents the virtual object location definer 100. A video capture processor 110 

1 8 processes video and audio content 36 on a frame by frame basis, converting the original content 

19 36 into a corresponding digitized representation. The processed content 36 ! is then stored in 

20 content buffer 120 for future access. A pre- viewer subsystem 130 allows for the viewing of a 

21 video frame of the processed content 36'. Frame N 141, for example, (shown in Figure 7) 

22 associated with the processed content 36', may be retrieved from the content buffer 120, viewed, 

23 and passed to a location selector processor 140. The location selector processor 140 allows for 

24 the selection of where in the frame N 141 the virtual object 38 may be placed. When the frame 

25 N 1 4 1 is retrieved by the location selector processor 1 40, either a static area may be selected, or 

26 alternatively, a dynamic area, which is tied to an area within the frame of the processed content 

27 36', may be selected. An overlay matte 16 (see Figure 7) may be used in the virtual object 

28 insertion process to identify where and how a virtual object location 37 is to be placed in the 

29 processed content 36'. 
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1 Techniques for pattern recognition used by the location selector processor 140 to 

2 facilitate the creation of the matte 1 6 and the identification of the pixels within the frame that the 

3 matte 1 6 is to be associated with for that frame are described in detail in US Patent 5,808,695, to 

4 Rosser, Roy J.; Das, Subhodev; and Tan, Yi; entitled Method of Tracking Scene Motion for Live 

5 Video Insertion; US Patent 5,903,317, to Sharir, Avi; and Tamir, Michael; entitled Apparatus 

6 and method for Detecting, Identifying, and Incorporating Advertisements in a Video; US Patent 

7 5,524,065, to Yagasaki, Toshiaki; entitled Method and Apparatus for Pattern Recognition; US 

8 Patent 5,627,915, to Rosser, Roy J.; Das, Subhodev; and Tan, Yi; von Kaenel, Peter; entitled 

9 Pattern Recognition System Employing Unlike Templates to Detect Obj ects Having Distinctive 

10 Features in a Video Field; and US Patent 4,817,171, to Stentiford, Frederick; entitled Pattern 

1 1 Recognition System, the disclosures of which are hereby incorporated by reference. 

12 When the area is selected by the location selector processor 140 and the overlay matte 16 

13 for the initial video frame N 141 is created, a video object marker processor 160 creates the 

14 transparent overlay matte 16 that is associated with the selected area for subsequent frames, for 

15 example frame N+l 142 and frame N+2 143 of the processed content 36 ! , for the duration of 

16 frames designated, as shown in Figure 7. This selected area defines the virtual object location 

17 37. Pattern recognition technology may then be applied to each subsequent frame of the 

1 8 processed content 36' in the video object marker processor 1 60, creating a sequence of mattes to 

19 be applied to each frame of the processed content 36 ! , moving and transforming as needed to 

20 match the temporal movement and transformations of the virtual object location 37 within the 

21 processed content 36' to which the virtual object 38 is to be tied. The pattern recognition 

22 technology handles transitions, cutaways, and cutbacks within the processed content 36', and any 

23 visual blocking or occlusions that may occur as other objects within the processed content 36* 

24 appear in front of the dynamic area selected for virtual object location 37. 

25 Simultaneously with the selection of the virtual object location 37 and the creation of the 

26 mattes, a virtual object rules processor 170 allows for the entry of rules that govern the types of 

27 virtual objects and other relevant placement guidelines associated with the virtual object location 

28 37. These rules allow for the selection of characteristics such as the duration of the virtual 

29 object location 37, and viewing overlay characteristics such as transparency of the overlay 

30 virtual object, and whether the virtual object location 37 is suitable for an interactive virtual 
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1 object. The operations center 1 0 processes the stored, non-realtime processed content 36' and the 

2 real-time (live) processed content 36 f . For real-time processed content 36' the content buffer 120 

3 serves as a short buffer, and predefined rules are pre-loaded into the virtual object rules 

4 processor 170. Additionally, the video object marker processor 160 is pre-loaded with the 

5 directions as to which locations within the processed content 36* are to be treated as virtual 

6 object locations. The video object marker processor 160 then automatically searches the 

7 real-time processed content 3 6' using pattern recognition technologies presented above, or other 

8 technologies, and automatically creates the mattes required for each virtual object location. 

9 Once the video object marker processor 160 creates the mattes and the associated controls, the 

10 mattes are associated with the actual processed content 36' in the content buffer 120. The 

11 processed content 36', along with the mattes are then optionally processed using the optional 

12 video processor 150, which performs any necessary content encoding (e.g., MPEG4, or 

13 digitalization), and makes the content 36' available to a rules application processor 180. The 

14 rales application processor 1 80 creates metadata packets that carry the virtual object placement 

15 rules information and mattes and associates these packets with the processed content 36' for each 

16 virtual object location 37 selected in the virtual object location definer 100. 

17 Figure 8 is a block diagram of the virtual object selector 200. Processed content 36', 

18 along with the metadata packets carrying the virtual object placement rules information 

19 associated with each virtual object location 37 and the mattes 16 are provided by the virtual 

20 object location definer 100 to the virtual object selector 200. An object selector processor 210 

2 1 extracts the placement rules and stores the processed content 36' in a content buffer 240. Using 

22 the placement rules, along with any operator entered object placement guidance, the object 

23 selector processor 210 queries an obj ect matcher processor 23 0 to initiate the selection of virtual 

24 objects that match the requisite rules. The object matcher processor 230 can be commanded by 

25 the object selector processor 210 to match a virtual object 38 in at least three manners: 1) 

26 automatically, 2) with manual placement, and 3) with pre-selected virtual objects. For automatic 

27 matching, the obj ect matcher processor 230 searches an available virtual obj ects database 220 to 

28 find virtual obj ects that meet the placement rules provided by the obj ect selector processor 210. 

29 The matching virtual objects are then marked in the available virtual objects database 220 as 

30 suitable for that virtual object location 37. For manual matching, the operator of the object 
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1 matcher processor 230 manually selects the desired virtual obj ects to be associated with a virtual 

2 object location 37, and marks the selected virtual objects as suitable for the virtual object 

3 location 3 7 in the available virtual obj ects database 220. For pre-selected obj ects, the placement 

4 rules will indicate the pre-defined virtual objects to be associated with the processed content 36'. 

5 The object matcher processor 230 marks the pre-determined virtual objects in the available 

6 virtual objects database 220 as being associated the particular processed content 36 f and virtual 

7 object location 37. 

8 Virtual obj ects may be processed and stored in the available virtual obj ects database 220 

9 before they are used. Processing of the virtual objects includes digitizing the virtual object 38 

1 0 and associating the virtual obj ect with those virtual obj ect 3 8 placement guidelines and rules that 

1 1 must be followed to place the virtual object 38 within virtual object locations. The rules and 

1 2 guidelines may include product categories with which the virtual obj ect 3 8 should be associated, 

13 or in contrast, cannot be associated with, the type of virtual object 38, the duration that the 

14 virtual object 38 is valid to be used, the number of times the virtual object 38 may be used, and 

15 whether the virtual object 38 is interactive and any interactive virtual object trigger action 56 or 

16 optional virtual object software applet 152 associated with an interactive virtual object 38. 

17 In a non-realtime environment, an optional post viewer processor 260, which is preceded 

18 by a virtual object insertion processor 250, is used to view the content 36 and insert each virtual 

19 object 38 that was matched to the content 36 by the object matcher processor 230 in the 

20 corresponding virtual object location 37, Techniques for insertion of overlaid virtual objects are 

21 described in detail in U.S. Patents 4,3 19,266 to Bannister, Richard S.; entitled Chroma Keying 

22 System; 4,999,709 to Yamazaki, Hiroshi; and Okazaki, Sakae; entitled Apparatus for Inserting 

23 Title Pictures; 5 ,249,03 9, to Chaplin, Daniel J. ; entitled Chroma Key Method and Apparatus; and 

24 5,233 ,423 to Jernigan, Forest E. ; and Bingham, Joseph; entitled Embedded Commercials within 

25 a Television Receiver using an Integrated Electronic Billboard, the disclosures of which are 

26 hereby incorporated by reference. 

27 Techniques for the insertion of embedded virtual objects are described in detail in U.S. 

28 Patents 5,953,076, to Astle, Brian; and Das, Subhodev; titled System and Method of Real Time 

29 Insertions into Video Using Adaptive Occlusion with a Synthetic Reference Image; 5,892,554, to 

30 DiCicco, Darrell; and Fant, Karl; entitled System and Method for Inserting Static and Dynamic 
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1 Images into a Live Video Broadcast; 5,515,485, to Luquet, Andre; andRebuffet, Michel; entitled 

2 Method and Device for Modifying a Zone in Successive Images; 5,903,3 1 7, to Sharir, Avi; and 

3 Tamir, Michael; entitled Apparatus and Method for Detecting, Identifying and Incorporation 

4 Advertisements in a Video; and the MPEG4 standard, the disclosure of which are hereby 

5 incorporated by reference. 

6 In a realtime environment, the optional post viewer processor 260 is bypassed, and the 

7 default virtual object 38 is placed in the virtual object location 37 by a default virtual object 

8 insertion processor 270, which includes (not shown) a virtual object insertion processor 250. 

9 The targeted virtual object management system (TVOMS) 300 shown in Figure 9 allows 

10 for virtual objects, including virtual object-based advertisements, to be directed to subscribers 

1 1 based on, for example, the use of subscriber data, programs watched data, past virtual objects 

1 2 viewing data, past interactive virtual obj ects selected data, and/or mood indicators entered by the 

13 subscriber. Alternatively, input from subscribers collected through form-based questionnaires 

14 (hard copy, electronic, and telephone, for example) may be used to further define a subscriber's 

15 potential likes, wants, and needs. Advertisers wanting to optimize their advertising expenditures 

16 may direct virtual objects to the appropriate viewing audiences to ensure that the desired 

17 audience views specific virtual objects. Specifically, advertisers can display specific virtual 

18 objects in content 36 that is being viewed by those subscribers most likely to be influenced to 

19 buy the advertised product, or otherwise respond in a desired fashion to the virtual objects. 

20 Virtual objects may also be targeted to reception sites on various levels. At a highest 

21 level, virtual objects can be delivered to all reception sites viewing content 36, with no targeting 

22 of the virtual objects to the subscriber, but with the virtual objects displayed in the content 36 

23 that are determined to be most relevant to the content 36. That is, the virtual objects are placed 

24 in the virtual object location 37 without the use of an individual or group targeting algorithm. 

25 Alternatively, some level of targeting may occur based on, for example, ADI, zip code +4, 

26 geographical data and other similar criteria known about a reception site 30. In this alternative 

27 embodiment, the virtual obj ects are sent to a reception site 30, and a local insertion routine in the 

28 reception site 30 controls placement of the virtual objects into the virtual object locations 37 in 

29 the content 36. The virtual obj ects may be stored at the reception site 30 and may be periodically 

30 refreshed. To account for reception sites that do not have virtual objects available for insertion, 
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1 the content 36 may be provided with a default virtual object 38 embedded in the content 36. 

2 Upon receipt of the content 36 at a reception site 30, the reception site 30, using the local 

3 insertion routine, determines if the default virtual object 38 should be replaced with another 

4 virtual obj ect 3 8 residing in the reception site ' s memory or being delivered concurrently with the 

5 content 36. 

6 Alternatively, virtual obj ects may be targeted to groups of reception sites, with the groups 

7 of reception sites categorized based on some other common subscriber characteristics such as 

8 programs watched data or interactive virtual objects selected data, for example. Finally, virtual 

9 obj ects may also be targeted to specific subscribers that share the use of a reception site 30 based 

10 on their unique subscriber characteristics. 

1 1 To target virtual objects, the TVOMS 300 may make use of information from numerous 

12 sources. These sources include collected programs watched data that are stored in the reception 

13 site 30, and periodically uploaded to the central data collection center 50 or the local data 

14 collection center 40, and from past virtual objects viewed information or past interactive virtual 

15 objects selected that is stored in the reception site 30 and periodically uploaded to the data 

16 collection centers. Additionally, these sources may include information from marketing 

17 databases and past television programs watched data, as described in U.S. Patent No. 5,798,785, 

18 entitled TERMINAL FOR SUGGESTING PROGRAMS OFFERED ON A TELEVISION 

19 PROGRAM DELIVERY SYSTEM, filed December 2, 1 993, incorporated herein by reference. 

20 The TVOMS 300 provides the management of information required to support each of 

21 the following; (1) delivery of targeted virtual objects along with content 36 being broadcast; (2) 

22 delivery of targeted virtual objects to subscribers independent of any content 36 being broadcast; 

23 and (3) delivery of TVOMS-related subscriber-specific information and commands. 

24 Figure 9 shows the TVOMS 3 00 supporting the targeting of virtual obj ects to subscribers. 

25 Broadcast information can be destined for the entire population of subscribers receiving the 

26 content 36, groups of subscribers, and individual subscribers. Broadcast information can include 

27 actual content 36, metadata packets with virtual object insertion control information, virtual 

28 objects for placement within the content 36, and command information required by the 

29 subscriber's reception site 30 to configure the reception site 30 and retrieval plans to guide the 

30 reception site 30 in placing the appropriate virtual object 38 within the content 36. Broadcasting 
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1 may be supported over a variety of broadcast-capable communication systems, such as the 

2 Internet, cable television systems, terrestrial broadcast systems, satellite broadcast systems, and 

3 wireless communications systems, and other systems described below. 

4 A subscriber information database 1210 contains subscriber information collected from 

5 numerous sources for each subscriber or reception site 30. The subscriber information may then 

6 be used by a virtual object targeting system 1220 to determine the best virtual objects to be 

7 distributed for inclusion in the content 36. Additionally, the information collected may be used 

8 to determine if the subscriber information has changed to the point that refreshed virtual obj ects 

9 should be delivered to a subscriber or, alternatively, whether a subscriber's group assignments 

10 should be updated. The virtual object targeting system 1220 determines the optimum subset of 

11 virtual objects to be associated with the content 36 based on the selected object metadata 

12 provided by the virtual object selector 200 (Figure 5) and subscriber information from the 

13 subscriber information database 1210. A content and virtual object packager 1260 is directed to 

14 retrieve the appropriate virtual objects from an available virtual objects database 1265. The 

15 content and virtual object package 1260 then, along with the content 36, from a content buffer 

16 1270, addresses the virtual objects with the appropriate group addressing information, and 

17 packages the virtual objects with the content 36. A delivery processor 1300 then delivers the 

18 combined package of virtual objects, content 36, and metadata to subscribers. 

19 As an alternative to delivering virtual objects with associated content 36, virtual objects 

20 can be delivered independently to individual subscribers or groups of subscribers based on 

21 updated subscriber information, modified group assignments, or the need for refreshed virtual 

22 objects at the reception site 30. Initiation could be automatic based on a scheduled cycle or by 

23 TVOMS operator direction. Upon delivery initiation, the virtual object targeting system 1220 

24 uses subscriber information from the subscriber information database 1210, information about 

25 available virtual objects from the available virtual objects database 1265, and information about 

26 previously delivered virtual obj ects from the subscriber information database 1 2 1 0, to select the 

27 appropriate virtual objects to be packaged and delivered to a reception site 30. Once the virtual 

28 object targeting system 1220 determines the appropriate virtual objects, the content and virtual 

29 obj ect packager 1 260 retrieves the appropriate virtual obj ects, packages the virtual obj ects with 

30 reception site configuration information, addresses the information either to a single subscriber 
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1 or group of subscribers, and delivers the information to the appropriate reception site 30 using a 

2 delivery processor 1 3 00. This delivery can be done in broadcast fashion or by communicating to 

3 reception sites directly. Virtual objects may be broadcast to all reception sites, and a reception 

4 site 30 may store only the virtual objects that are associated with groups to which the reception 

5 site 30 belongs. Alternatively content 36, virtual objects, and other information destined to 

6 reception sites may be provided to the object delivery center 15 (Figure 1) for delivery to 

7 reception sites. 

8 The databases addressed in Figure 9 may be configured to support a variety of 

9 information necessary for the TVOMS 300 to manage the targeting process. Below are tables 

10 that present typical data that may be tracked by these individual databases. 

11 Subscriber Information Database 1210 

12 Reception system identification information 

13 Reception site type 

1 4 Date of system set-up 

15 Date of last communication with operations center 

16 Household income 

17 User data (for each registered subscriber), including: 

18 Name 

19 Sex 

20 Age 

21 Place of birth 

22 . Education 

23 Profession 

24 TV program preferences 

25 Demographic information 

26 Past advertising viewed data, which virtual objects, time spent viewing, 

27 Past products ordered, along with time, date, and method of order 

28 Past billing information 

29 Imputed subscriber data from marketing databases 

30 Past TV programs watched data, along with time and date 



17 



5221 



1 Past PP V programs ordered data, along with time and date 

2 Mood indicators 

3 Form based questionnaire results 

4 Communication methods available (available options for both return and 

5 delivery) 

6 Group assignments per subscriber for each category 

7 Past virtual objects delivered to subscriber, date of delivery, method of delivery 

8 Past selected interactive virtual objects 

9 Zip+4 information 

10 Available Virtual Objects Database 1265 

11 Virtual object identifier with actual digital version of virtual object Display 

12 options (e.g., text, audio, graphics, video, link, HTML, XML, interactive) 

13 Static vs. dynamic virtual object indicator, 

14 If an interactive virtual object, interactive virtual object trigger action 

15 information 

16 If an interactive virtual obj ect, optional interactive virtual obj ect software applet 

17 Pricing subsidy information 

18 Run through completion status mode indication 

19 Date of valid use 

20 Virtual object placement controls, acceptable frequency 

21 Category and group preferences (as virtual object ranking percentages) 

22 Pending Commands Database 1215 

23 For each pending command: 

24 Destination address 

25 Actual command 

26 Date generated 

27 Date of confirmed receipt 

28 Within the TVOMS 300, the virtual object targeting system 1220 is responsible for the 



29 intelligent and rapid selection of virtual obj ects for placement in content 36. Category and group 
3 0 targeting is managed in a manner similar to that described in co-pending U. S . Application Serial 
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1 No. 09/597,893 entitled METHOD AND APPARATUS FOR TARGETING VIRTUAL 

2 OBJECTS, filed June 19, 2000, and in co-pending U.S. Application Serial No. 09/054,419 

3 entitled TARGETED ADVERTISEMENT USING TELEVISION DELIVERY SYSTEM, filed 

4 April 3, 1 998, and in co-pending U.S. Application Serial No. 09/328,672 entitled ELECTRONIC 

5 BOOK SELECTION AND DELIVERY SYSTEM WITH TARGETED ADVERTISING, filed 

6 on June 9, 1 999, each of which are incorporated herein by reference. 

7 Careful management of the virtual objects within the content 36, based on information 

8 known about the demographics and viewing habits of subscribers, for example, can greatly 

9 increase both the advertisers' likelihood of reaching an interested subscriber, and the likelihood a 

10 subscriber will be interested in a specific virtual object 38. Each virtual object location 37 

11 within the content 36 is assigned a series of virtual objects by the TVOMS 300, and when 

12 multiple virtual objects are delivered for a given virtual object location 37 in the content 36, a 

13 retrieval plan is developed that directs which virtual objects should be displayed for a given 

14 subscriber or reception site 30, a group of subscribers or reception sites, or the entire subscriber 

15 population. 

16 The process of managing the targeted virtual objects may consist of a number of 

17 configuration and set-up steps shown in Figure 1 0 that begins with the start step shown in block 

18 7010 and ends with the end step shown in block 7017. First, individual reception site address 

19 information is collected by a subscriber data collection engine 1202 in the address information 

20 collection block 7011. This address information uniquely identifies each reception site 30 

2 1 subscriber and associates necessary address information about each subscriber with the reception 

22 site identifier to aid in the virtual objects targeting process. This address information includes 

23 subscriber profile information, programs viewed information, past virtual obj ects delivered and 

24 viewed, and responses to menu-based questionnaires or other questionnaires completed by the 

25 subscriber. In block 7012, other subscriber information maybe collected from various sources, 

26 including surveys and marketing databases correlated by address or zip code+4, for example. 

27 Next, a number of target categories are defined as shown in block 7013. Examples of 

28 target categories include demographic targeting (age/sex/income) and location, such as Area of 

29 Dominant Influence (ADI). Next, as shown in block 7014, each target category is then 

30 segmented into appropriate groups. For example, the ADI may include Los Angeles, CA and 
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1 Washington D.C. New target categories can be added and the groups comprising the target 

2 category redefined after their initial establishment. 

3 Next, as shown in block 701 5, for each target category, each reception site 30 is assigned 

4 to a group based on the information collected about the subscriber. Once each subscriber is 

5 assigned to a group, the group assignments are conveyed to the reception site 30 and stored 

6 therein, as shown in block 7016. As groups are modified or group assignments change, the 

7 reception sites are provided with the changes. Additionally, the group assignment information is 

8 periodically resent to the reception sites to ensure that newly added reception sites and those 

9 reception sites that have accidentally lost their information are up-to-date. Alternatively, the 

10 reception site 30 may perform the processing of information about the characteristics of the 

1 1 subscriber, and generation of the group assignment information internal to the reception site as 

12 presented in co-pending U.S. Application Serial No. 09/628,805 entitled METHOD AND 

13 APPARATUS FOR LOCALLY TARGETING VIRTUAL OBJECTS WITHIN A TERMINAL, 

14 filed July 28, 2000, which is incorporated herein by reference. 

15 Returning to Figure 9, the virtual object targeting system 1220 determines the optimum 

16 types of virtual objects to be placed in the content 36 from the selected virtual objects provided 

1 7 by the virtual obj ect selector 200 (Figure 5), The virtual obj ect targeting system 1 220 takes into 

18 account subscribers who will likely view the content 36, the desirability of providing available 

19 virtual objects to those subscribers, target categories, the number of virtual objects locations 

20 available for the content 36, and the number of virtual objects available for assignment for a 

2 1 given virtual obj ect location 3 7 . 

22 Once specific virtual objects are selected for one or more available virtual object 

23 locations 37, the groups that should view each virtual object 38 are determined, based on the 

24 target category of interest. The selected virtual object locations 37 may include all virtual object 

25 locations, or a subset of all the virtual object locations. A retrieval plan is generated by the 

26 retrieval plan generator 1275 that provides information concerning which target category and 

27 groups are assigned to each virtual object 38 associated with each virtual object location 37. The 

28 retrieval plan may provide information for one virtual object location 37 or multiple virtual 

29 object locations within content 36, where one or more virtual objects, target categories, and the 

30 groups to which each virtual object 38 is targeted within each virtual object location 37 is also 
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1 provided. An example retrieval plan is provided in Table C below. Alternatively, the retrieval 

2 plan providing virtual obj ect assignments to virtual obj ect locations may be sent independently 

3 from the retrieval plan providing virtual obj ects, target categories, and the groups to which each 

4 virtual object 38 may be targeted. Retrieval plans may be distributed along with the virtual 

5 objects and the associated content 36 directly to the reception sites by the delivery processor 

6 1 3 00 or using the obj ect delivery center 1 5 . Alternatively, a retrieval plan may be distributed by 

7 the delivery processor 1 3 00 or using the obj ect delivery center 1 5 independent of the associated 

8 content 36 or virtual objects. 

9 After the reception site 30 receives and stores the virtual objects and the retrieval plan, 

10 the reception site 30 inserts those virtual objects into the appropriate virtual object locations in 

11 the content 36 based on the retrieval plan. The reception site 30 may retrieve and store only 

1 2 those virtual obj ects associated with that reception site's group assignment for that virtual object 

13 location 37. Alternatively, the reception site 30 may retrieve and store all virtual objects but 

14 only insert those virtual objects into virtual object locations as dictated by the retrieval plan. 

15 When the virtual objects are displayed within the content 36, the reception site 30 will 

16 store virtual objects viewed data indicating that a virtual object 38 was shown. In an 

17 embodiment, the reception site 30 will store this virtual object viewed data only if the virtual 

18 objects are displayed for a predetermined time, or only if the subscriber takes an action to 

19 indicate the virtual object 38 has been viewed, such as by selecting an interactive virtual object 

20 38, for example. Accumulated virtual objects viewed data may be collected from a reception site 

21 30 at a later time for review purposes. Unique reception site identification information also may 

22 be provided with the collected virtual obj ects viewed data. Upon collection of the virtual obj ects 

23 viewed data, the reception site 3 0 may return the used memory space to available pools for future 

24 use. 

25 The virtual object targeting system 1220 receives requests from the metadata extractor 

26 processor 1200 to initiate the determination of virtual objects to be placed. The metadata 

27 extractor processor 1 200 receives content 3 6 and associated virtual obj ect information from the 

28 virtual object selector 200 (Figure 5). The virtual object targeting system 1220 provides outputs 

29 to the content and virtual object packager 1260 and the retrieval plan generator 1275. 



21 



5221 



1 A part of the TVOMS 300 operation is the retrieval of subscriber data, and the 

2 assimilation of the subscriber data into the virtual objects selection method. This operation 

3 typically includes two steps. First, subscriber data is retrieved from the reception sites by the 

4 central data collection center 50 or the local data collection center 40 (Figure 1). The subscriber 

5 data is compiled and sent to the data collection engine 1202 in the operations center 10. Once 



6 assembled at the TVOMS 300, the data is filtered for each application of the TVOMS 300. In an 

7 embodiment, the subscriber information database 1210 receives inputs from the subscriber data 

8 collection engine 1202 and a configuration set-up system 1205. The subscriber information 

9 database 1210 provides outputs to the configuration set-up system 1205, and the virtual object 
10 targeting system 1220. 



1 1 The data gathered includes: 

12 What products a subscriber purchased and when they were purchased, 

13 What Pay Per View (PPV) TV programs a subscriber purchased and when they 

14 were purchased, 

15 What television programming a subscriber has viewed, 

16 What interactive virtual objects have been selected, 

17 What virtual objects a subscriber viewed and for how long, and 

1 8 Subscriber profile information. 

1 9 Subscriber profile information may be collected and stored for one or more subscribers 

20 for the purposes of virtual objects targeting. The subscriber profile may include demographic 

21 information that may be gathered in a number of ways. The reception site 30 builds the 

22 subscriber profile for each subscriber and stores the information in a memory file by subscriber 

23 name. The file may be uploaded to the central data collection center 50 or the local data 

24 collection center 40 and provided to subscriber data collection engine 1202 periodically. 

25 Subscriber preference information may be collected using on screen menus at the reception site 

26 30, including information such as name, sex, age, place of birth, place of lower school education, 

27 employment type, level of education, amount of television program viewing per week, and the 

28 number of television shows in particular categories that the subscriber watches in a given week 

29 such as, sports, movies, documentaries, sitcoms, amount of Internet use and favorite web sites, 
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1 etc. Any demographic information that will assist the TVOMS 300 in targeting virtual objects 

2 may be used. 

3 In addition to demographic information gathered at the reception site 30, the subscriber 

4 profile can be compiled using other methods. For instance, subscriber information can be 

5 gathered using questionnaires sent by mail and subsequently entered in the subscriber 

6 information database 1210. 

7 As an alternative to gathering demographic data, a simulated subscriber profile can be 

8 generated using an algorithm that analyzes subscriber access history and subscriber habits. 

9 Using test information generated from a statistically significant number of subscribers, the 

10 simulated subscriber profile algorithm estimates the subscriber's age, education, sex and other 

1 1 relevant information. The analysis then compares information about the subscriber, for example 

12 the subscriber's programs watched information, with that of the test group. An example of the 

13 type of information maintained for a subscriber profile is presented below. 

14 The subscriber profile data fields are an example of typical fields that can be used in the 

15 databases. Definitions of various fields are listed below. The primary purpose of profiling the 

16 subscriber is to acquire marketing information on the subscriber's likely response to available 

17 virtual objects. Ancillary information may be available including actual program selections or 

18 interactive virtual objects selections. Information tracked within the subscriber's profile 

19 includes: 

20 Subscriber ID A unique identifier generated by the system, one for each 

21 subscriber using a specific reception site. 

22 Reception site types Boolean field that identifies the type of reception site 

23 used. 

24 Reception site ID ID of the reception site. 

25 Hookup Date Date physical hardware is connected. 

26 A demographic profile may be constructed for each subscriber from questionnaires or 

27 other sources. The following fields represent this demographic information: 

28 Subscribers Age 2-5 Boolean field if the household has subscribers between 2 

29 and 5 years of age. 
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1 Subscribers Age 6-1 1 Boolean field if the household has subscribers between 6 

2 and 1 1 years of age. 

3 Subscribers Age 12-17 Boolean field if the household has subscribers between 12 

4 and 1 7 years of age. 

5 Subscribers Age N1-N2 Boolean field if household has subscribers between Nl 

6 and N2 years of age. 

7 Income Annual household income. 

8 Zip Code+4 Self-explanatory. 

9 Occupancy Number of subscribers in household. 

10 Highest Education Highest level of education of any subscriber in the 

1 1 household. 

12 Field of Use Personal, professional, educational, other. 

13 Profession Self-explanatory. 

14 Education Level Self-explanatory. 

15 These subscriber profile inputs may assist in the assignment of reception sites to groups 

1 6 for each target category. There are numerous variations to the field definitions listed above, such 

17 as different age groupings, for example. Other subscriber profile data fields may also be 

18 specified. 

1 9 Marketing information, such as the demographics of subscribers, may be received from a 

20 central data collection center 50, a local data collection center 40, other external sources, or 

2 1 directly from the reception sites using the subscriber data collection engine 1 202. To effectively 

22 manage the virtual obj ects targeting operations, marketing information, such as the existence of 

23 markets for certain products, may be provided to the TVOMS 300. The following examples of 

24 information may be maintained in the subscriber information database 1210: subscriber 

25 demographic profile, subscriber buy information, and correlation of demographic information 

26 with buy information. The subscriber data collection engine 1202 gathers the marketing 

27 information from the various sources and indexes the information for inclusion in the subscriber 

28 information database 1210. 

29 To maintain the subscriber information database 1210 within the TVOMS 300, a 

30 database server 1 1 90, communications server 1191, subscriber workstation 1 1 92 or stations, or 
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1 the suitable equivalents thereof, may be used, as depicted in Figure 11. The database server 

2 1 190 supports saving database files, event logging, event scheduling, database server services, 

3 and database security access. 

4 The communications server 1191 performs the following functions on database data: 

5 integrity check, filtering, processing, downloading to reception sites using the pending 

6 commands database 1215, and uploading subscriber data from reception sites using the 

7 subscriber data collection engine 1202. The subscriber workstation 1 192 allows for operator 

8 viewing and entry of subscriber data into the subscriber information database 1210. 

9 Figure 12 shows an example of the configuration set-up system 1205 in more detail. An 

10 interface 1206 receives individual addressing information unique to reception sites. The 

1 1 interface 1 206 can include a workstation, such as the workstation 1 209, for example, from which 

12 an operator manually enters reception site information. Alternately, reception site information 

1 3 can be automatically entered at the interface 1 206 by downloading from an off-site database, the 

14 Internet, a storage medium, such as a CD-ROM or a floppy disk, or by collecting the information 

15 directly from the individual reception sites using the subscriber data collection engine 1202 or 

16 provided by a central data collection center 50 or local data collection center 40. A processor 

17 1207 processes the received reception site information and organizes the information for use. 

1 8 For example, the processor 1207 may create a Category/Group Definition Matrix as presented in 

19 Table A and a Group Assignment Matrix as presented in Table B that can be used to target 

20 virtual objects to groups of reception sites or to an individual reception site 30. In an alternative 

21 embodiment, if subscriber information is available where multiple subscribers may share a 

22 reception site 30, a Group Assignment matrix may be created for each subscriber who shares the 

23 reception site 30. The Category/Group Definition Matrix and Group Assignment matrices will 

24 be described in more detail later. The Category/Group Definition and Group Assignment 

25 matrices and organized reception site information are then stored in a database 1208, and are 

26 periodically updated as reception site information, for example, changes. 

27 The information used by the processor 1207 to create a database of the Category/Group 

28 Definition and Group Assignment matrices includes, for example, the reception site identifier, 

29 subscriber identifier, zip code + 4 data, household income, and age and sex of the subscribers, 

30 for example. The information gathered by the configuration set-up system 1 205 can come from 
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1 a variety of sources including marketing databases, direct inputs from the subscribers, data 

2 collected by the subscriber data collection engine 1202, a central data collection center 50, a 

3 local data collection center 40, and other sources. The processor 1207 will assign category 

4 numbers to target categories. For example, the ADI could be assigned category 1 and household 

5 (HH) income could be assigned category 2. Next, the configuration set-up system 1205 creates a 

6 number of non-overlapping groups for each category. For example, ADI can be broken down 

7 into Seattle, WA, Washington D.C., Denver CO., Los Angles CA, etc. Similarly, HH income 

8 can be broken down into a number of income groups such as no income, 20-40K, 40-60K, 

9 60-120K, and over 120K. Then, the configuration set-up system 1205 assigns a "group mask 

10 representation" for each group within every category. The group mask representation may be 

1 1 simply a binary number that can be used to identify a particular group. Table A shows a 

12 completed Category/Group Definition matrix that could be used by the virtual object targeting 

13 system 1220 to assign targeted virtual objects to groups of reception sites or to individual 

14 reception sites. 

15 Table A - Category/Group Definition Matrix 



Category 
Number 


Category 
Name 


Group 
Number 


Group Definition 


Group Mask 
Representation 






1 


Seattle, WA 


1000000000 


1 


ADI 


2 


Washington, D.C. 


0100000000 






3 


Denver, CO 


0010000000 






4 


Los Angeles, CA 


0001000000 






1 


No income 


1000000000 


2 


HH income 


2 


20-40K 


0100000000 






3 


40-60K 


0010000000 






4 


60-120K 


0001000000 






1 


Group a 


1000000000 






2 


Group b 


0100000000 


3 


Category x 


3 


Group c 


0010000000 






4 


Group d 


0001000000 






5 


Group e 


0000100000 






6 


Group f 


0000010000 
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1 The processor 1207 also creates the Group Assignment matrix. The Group Assignment 

2 matrix, shown in Table B, assigns to each reception site 30, for each category, its corresponding 

3 group number. Associated with each group number is the group definition and the group mask 

4 representation. For example, the reception site 30 identified by the address 123 1 1 is assigned 

5 group number 2 (i.e., Washington D.C.) for ADI, and group number 3 (i.e., 40-60K) for 

6 household income. The Group Assignment matrix is updated periodically as categories and 

7 group definitions change, and as data related to individual reception sites or groups of reception 

8 sites change. Many other ways of organizing the information in a database for later use are 

9 possible. 

10 The configuration set-up system 1205 also delivers the group configuration (i.e., 

1 1 information specific to an individual reception site 30, from the Group Assignment matrix) to 

12 Table B Group Assignment Matrix 



Address 


Target Category 


Group 
Number 


Group Definition 


Group Mask 
Representation 




ADI 


2 


Washington, D.C. 


01000000000 


12311 


HH income 


3 


40-60K 


00100000000 




Category x 


5 


Group d 


00100000000 




ADI 


4 


LA 


00100000000 


12312 


HH income 


3 


60-1 20K 


00100000000 




Category x 


2 


Group a 


10000000000 


12313 


ADI 


3 


Denver 


00100000000 


HH income 


4 


60-80K 


00010000000 




Category x 


3 


Group b 


01000000000 



13 each reception site 30. For example, the reception site 30 assigned the address 123 1 1 is sent for 

14 category 1, group mask representation 01000000000, indicating group 2 assignment. 

15 The group configuration information can be stored in the pending commands database 

16 1215 to be transmitted directly to each reception site 30 periodically or the next time the 

17 reception site 30 establishes communications operations center 10. Each time a group 
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1 configuration message is generated, the message is stored in the pending commands database 

2 1215. 

3 Alternatively to the T VOMS 300 assigning the reception site 3 0 to individual groups for 

4 each category, the TVOMS 300 could deliver the group definitions and category definitions to 

5 the all reception sites. Each reception site 30 could then assign itself to the appropriate groups 

6 for each category based on internal processing algorithms. 

7 Figure 13 shows an embodiment of the virtual object targeting system 1220 in more 

8 detail. A resource management engine 1305 uses information from a metadata extractor 

9 processor 1200 and an available virtual object database 1265 (see Figure 9) to determine the 

10 number of virtual objects to be assigned to a given virtual object location 37. A virtual object 

1 1 placement engine 1307 decides which virtual objects to place in virtual object locations in the 

12 content 36. A group assignment engine 1 309 determines which reception sites will view specific 

1 3 virtual obj ects . The virtual obj ect placement engine 1307 receives information from the resource 

14 management engine 1305 related to the number of virtual objects available, how many virtual 

15 objects are to be provided for a given virtual object location 37, and the actual type of virtual 

16 objects available. 

17 The resource management engine 1 305 functions to divide available delivery bandwidth 

18 among multiple virtual objects for a given virtual object location 37 in the content 36. Because 

1 9 there may be a limited amount of resources on the delivery network 1 1 to deliver virtual obj ects 

20 with the content 36, the resource management engine 1 305 may assign the available bandwidth 

2 1 optimally for the virtual obj ects associated with the individual virtual obj ect locations within the 

22 content 36 being delivered over the communication channels. Some virtual object locations may 

23 be assigned multiple virtual objects, each targeted to a different group or groups, whereas other 

24 virtual object locations may be assigned only a single virtual object 38. 

25 Referring to Table A, four group numbers (i.e., 1-4) are shown for the category of 

26 targeted virtual objects, ADL For a particular virtual object location 37 in the content 36, the 

27 four groups can be divided into two, one for each available virtual object 38 of two total, with 

28 groups 1 and 2 receiving virtual object A and groups 3 and 4 receiving virtual object B, as shown 

29 for virtual object location 1 . A retrieval plan for this later example is shown in Table C. 

30 
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1 Table C - Retrieval Plan 



Virtual Object 
Location 


Target 
Category 


Virtual Object To 
Retrieve 


Groups Assigned to 
Specific Virtual Object 


Group Mask 
Assignment 


Virtual Object Location 1 


AD I 


Virtual Object A 


1 o 

1,1 








Virtual Object B 


3,4 


00110000000 


Virtual Object Location 2 


HH Income 


Virtual Object A 


1,2,3 


11100000000 






Virtual Object B 


4 


00010000000 


Virtual Object Location 3 


Category x 


Virtual Object A 


1,2 


11000000000 






Virtual Object B 


3 


00100000000 






Virtual Object C 


4 


00010000000 






Virtual Object D 


5 


00001000000 






Virtual Object E 


6 


00000100000 


Virtual Object Location 4 


All 


Virtual Object A 


All 


11111111111 



2 After determining how many virtual objects will be needed for each virtual object 

3 location 37 within the content 36, the resource management engine 1305 may also account for 

4 the type of available targeted virtual objects for display and the variety of subscribers (according 

5 to group assignment numbers) who may be viewing the content 36. An advertiser or content 

6 provider may provide this information when forwarding virtual objects for insertion. 

7 In an embodiment, the virtual object placement engine 1 307 determines which specific 

8 virtual objects are to be placed in each available virtual object location 37 within the content 36. 

9 The virtual object placement engine 1307 first receives the list of selected available virtual 

10 objects from the metadata extractor processor 1200 (Figure 9). In cooperation with the resource 

11 management engine 1305 ? the virtual object placement engine 1307 then determines which of the 

12 available virtual objects should be placed in each virtual object location 37 within the content 36. 

13 For example, if the preferred target category for virtual object location 1 is ADI, the virtual 

1 4 obj ect placement engine 1 3 07 will select one or more targeted virtual obj ects determined by the 

1 5 metadata extractor processor 1 200 to place in that virtual obj ect location 37. If the demographic 
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1 or other data assembled by the configuration set-up system 1205 indicates that more than one 

2 targeted virtual object 38 should be placed, depending on the ADI, then the virtual object 

3 placement engine 1307 will select the appropriate number of targeted virtual objects, and will 

4 assign each targeted virtual object 38 to the specific virtual object location 37. The operation of 

5 the virtual object placement engine 1 307 to assign the targeted virtual objects will be described 

6 in more detail later. 

7 In an embodiment, the group assignment engine 1 309 receives inputs from the resource 

8 management engine 1305 and the virtual object placement engine 1307 and then determines 

9 which reception sites and target category groups will view specific targeted virtual objects. 

10 Thus, for each virtual object location 37, the group assignment engine 1309 assigns the reception 

1 1 sites to one of the virtual objects. The reception sites can be assigned based on their placement 

12 within a group (i.e., based on their group assignment number) or based on their individual 

13 reception site unit address. In tables B and C, the assignments are shown based on the group 

14 assignment numbers. As also shown in Table C, the group addressing for a virtual object 

1 5 location 3 7 may be based on a single category of targeting. This may avoid a conflict regarding 

16 which virtual object 38 a reception site 30 may retrieve. 

17 The group assignment engine 1309 provides an output to the retrieval plan generator 

18 1275. The output indicates which group assignment numbers (i.e., which groups of reception 

19 sites) are assigned to a virtual object 38 for a given virtual object location 37 in the content 36. 

20 The retrieval plan generator 1275 then generates a bit word, or group mask assignment, that is 

21 used to assign the groups to virtual objects. Once generated, the retrieval plan is provided to the 

22 delivery processor 1 300 for distribution along with the content 36 and the actual virtual objects 

23 to reception sites by object delivery center 15. 

24 In an embodiment, the virtual object targeting system 1220 provides a virtual object 

25 generation request command 1261 to the content and virtual object packager 1260. The virtual 

26 objects generation request command 1261 specifies which particular virtual objects are to be 

27 displayed in a particular virtual object location 37, and the actual location of the virtual objects. 

28 The virtual object 38 is then retrieved from the available virtual object database 1265. The 

29 virtual objects, along with the retrieval plan, and content 36 and associated metadata packets are 

30 provided to the delivery processor 1300 for delivery to the appropriate reception sites. 
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1 When a reception site 30 receives the content 36 that contains targeted virtual objects, 

2 software instructions operating on the reception site 30 analyze the contents of the retrieval plan. 

3 Then, based on the groups assigned for each virtual object 38, the reception site 30 retrieves 

4 those virtual obj ects that match its own group assignments for the target category being used for 

5 the virtual object location 37. The reception site 30 then associates those virtual objects 

6 retrieved with the appropriate virtual object location 37 where the virtual object 38 will be 

7 placed, so that when the content 36 is viewed, the virtual object 38 assigned to that virtual object 

8 location 37 is displayed. 

9 An embodiment of the process for assigning targeted virtual objects using the virtual 

10 object placement engine 1307 is presented in Figure 14. The process begins with block 2360. In 

1 1 block 2362, the virtual object placement engine 1 307 assigns reception sites to groups. In block 

12 2364, the virtual object placement engine 1307 ties or relates virtual object locations in content 

13 36 to the groups. In block 2366, the virtual object placement engine 1307 ties or relates virtual 

14 objects to groups. In block 2368, the virtual object placement engine 1307 determines how 

15 many virtual objects to assign to a virtual object location 37. In block 2370, the virtual object 

16 placement engine 1307 determines which target category to use for one or more virtual object 

17 locations37. Inblock2372, the virtual object placement engine 1307 determines specific virtual 

18 objects to be placed in the virtual object locations 37. In block 2374, the virtual object 

19 placement engine 1307 determines which groups to assign to the virtual objects 38 for the 

20 selected virtual object locations 37. The process ends with block 2376. 

2 1 As discussed above, virtual obj ect targeting uses target categories and groups within each 

22 target category to tie or relate three entities together: 1) the reception site 30; 2) virtual objects; 

23 and 3) virtual object locations in content 36. In one embodiment of block 2362 in Figure 14, the 

24 reception sites are assigned to groups for each target category by the configuration set-up system 

25 1 205 based on numerous factors as described below. One method to assign the reception sites to 

26 groups is to use the zip code+4 as an index into one of the available demographic marketing 

27 databases. From the zip code+4 data, a distinct demographic cluster can be determined. The 

28 demographic cluster can then be mapped directly to the specific group within each target 

29 category. Manual assignment of groups to reception sites would be a daunting task for a large 

30 population of reception sites (approaching several million). Therefore, the processor 1207 in the 



31 



5221 



1 configuration set-up system 1205 may perform this function automatically, using its installed 

2 software routines. Alternative methods can also be devised to automatically map individual 

3 reception sites to groups within target categories. Once each reception site 30 is mapped to one 

4 group for each target category, the group assignments may be delivered to the reception site 30 

5 for storage. 

6 In one embodiment of block 2364 in Figure 14, virtual object locations in content 36 are 

7 tied or related to groups as described below. For each virtual object location 37, a group 

8 breakdown percentage can be defined for each group that represents the likely compatibility of 

9 the content 36 surrounding that virtual object location 37 with each group. Breakdown 

10 percentages for each virtual object location 37 are defined within the virtual object selector 200 

11 (see Figure 8) and passed to the TVOMS 300. Table D shows a sample breakdown of these 

1 2 group breakdown percentages for five example virtual obj ect locations for three example target 

13 categories. 

14 The group breakdown percentage data may be derived from a number of sources 

15 including surveys, ratings services, and virtual objects viewed data collected by the reception 

1 6 sites, for example. In this example, the three target categories are the same as those presented in 

17 Table B, and the group assignment numbers are the same as those presented in Table A. Thus, 

1 8 target categories 1 and 2 each have four groups associated with them, and target category 3 has 

19 six groups associated with it. For virtual object location 1, the target category 1 refers to ADI 

20 and under group 1 , a group breakdown percentage of 25 percent is assigned for group 1 from the 

21 target category ADI since 25 percent of the subscribers reside in the Seattle, WA ADI. The 

22 group breakdown percentages for each target category for each virtual object location 37 may 

23 sum to 100 percent. 

24 In an embodiment of the subroutine represented by block 2366 of Figure 14, virtual 

25 obj ects may be ranked according to their potential revenue generation for each group within one 

26 and up to all possible target categories, again using percentages. This information may be 

27 provided by an advertiser, programmer, or content provider responsible for the virtual objects 

28 and may reside in the available virtual objects database 1265. Table E shows a sample 

29 assignment of virtual obj ect ranking percentages for eight sample virtual obj ects using the same 

30 target categories and group numbers as in Table D. Not all virtual objects may be assigned to 
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1 groups for a target category if an advertiser or programmer does not wish its virtual obj ects to be 

2 targeted in the manner required by that target category. For example, an advertiser or 

3 programmer may want the same virtual object to be displayed at all reception sites 30, regardless 

4 of subscriber group information or characteristics. 

5 Table D - Virtual Object Location Group Breakdown Percentages 



Virtual 

UUJcll 

location 


Target 

v^dlcgui y 


Group 
i 


Group 
2 


Group 
3 


Group 
4 


Group 
5 


Group 
6 


Virtual 
object 
location 1 


1 


25 


25 


25 


25 


N/A 


N/A 


u 


2 


30 


10 


20 


40 


N/A 


N/A 




3 


10 


20 


30 


40 


N/A 


N/A 


Virtual 
object 
location 2 


1 


10 


20 


30 


40 


N/A 


N/A 


« 


2 


25 


25 


25 


25 


N/A 


N/A 




3 


10 


15 


25 


25 


15 


10 


Virtual 
object 
location 3 


1 


40 


30 


20 


10 


N/A 


N/A 


« 


2 


80 


10 


5 


5 


N/A 


N/A 




3 


25 


25 


10 


10 


15 


25 


Virtual 
object 
location 4 


1 


50 


0 


50 


0 


N/A 


N/A 


a 


2 


0 


40 


40 


20 


N/A 


N/A 


a 


3 


10 


10 


25 


25 


15 


15 


Virtual 
object 
location 5 


1 


20 


30 


30 


20 


N/A 


N/A 


u 


2 


30 


30 


10 


30 


10 


10 


a 


3 


10 


30 


10 


30 


10 


10 
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Referring to Table E, the data indicates that for virtual object 1, and target category 1 
(ADI), the advertiser believes that virtual object 1 is appropriate for the subscribers in groups 1 
and 2 and is not appropriate for the subscribers in groups 3 and 4. The advertiser also believes 
that virtual object 1 is equally appropriate for both the group 1 and the group 2 subscribers. 
However, if the group 1 subscribers are determined to be more likely to respond to virtual object 
1 than the group 2 subscribers, then group 1 could be given a higher percentage than group 2. 
Table E also shows that virtual object 1 is not applicable to groups 5 and 6 because only four 
groups are defined for the target category ADI. Thus, all the reception sites will be grouped into 
one of groups 1 through 4. 



Table E - Virtual Object Ranking Percentages 



Virtual Object 


Target 
Category 


Group 
1 


Group 
2 


Group 
3 


Group 
4 


Group 
5 


Group 
6 


Virtual object 1 


1 


50 


50 


0 


0 


N/A 


N/A 


u 


2 


30 


10 


20 


40 


N/A 


N/A 


u 


3 


0 


0 


0 


0 


0 


0 


Virtual object 2 


1 


0 


0 


50 


50 


N/A 


N/A 




2 


0 


0 


0 


0 


N/A 


N/A 


a 


3 


0 


0 


0 


0 


0 


0 


Virtual object3 


1 


0 


0 


0 


0 


N/A 


N/A 


u 


2 


25 


25 


25 


25 


N/A 


N/A 


u 


3 


0 


0 


0 


0 


0 


0 


Virtual object 4 


1 


50 


0 


50 


0 


N/A 


N/A 


« 


2 


0 


40 


40 


20 


N/A 


N/A 




3 


10 


30 


10 


30 


10 


10 


Virtual object 5 


1 


40 


20 


20 


40 


N/A 


N/A 




2 


10 


30 


30 


30 


N/A 


N/A 




3 


30 


30 


30 


5 


5 


0 


Virtual object 6 


1 


0 


0 


0 


0 


N/A 


N/A 


u 


2 


0 


0 


0 


0 


N/A 


N/A 


it 


3 


10 


10 


10 


10 


30 


30 


Virtual object 7 


1 


20 


40 


40 


20 


N/A 


N/A 


a 


2 


25 


25 


25 


25 


N/A 


N/A 


« 


3 


0 


30 


20 


30 


0 


20 


Virtual object 8 


1 


30 


40 


0 


30 


N/A 


N/A 


« 


2 


30 


30 


10 


30 


N/A 


N/A 




3 


20 


0 


20 


20 


20 


20 



Using this paradigm, virtual objects can be targeted using at least two methods. The 
first is a designated multi-virtual object campaign where specific unique sets of groups are 
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1 assigned for each virtual object 38 of the campaign. In the second method, each virtual object 38 

2 provided by an advertiser is independently associated with groups. Virtual obj ects from several 

3 different advertisers are then used together to optimize use of virtual object locations. 

4 As depicted in Figure 14, blocks 2368, 2370, 2372, and 2374, the virtual object 

5 placement engine 1307 determines: 1) how many virtual objects are assigned to which virtual 

6 object location; 2) which target category is used for which virtual object location; 3) which 

7 virtual objects to place in each virtual objects location; and 4) which groups are assigned to 

8 which virtual objects, respectively. To limit the need for excessive distribution bandwidth to 

9 distribute virtual objects to reception sites, the algorithm in the virtual object placement engine 

10 1307 that assigns targeted virtual objects to the virtual objects assumes that there is a total 

1 1 number of virtual objects available [TOTALJVIRTUAL OBJECTS] for a segment of content 36 

12 (across all virtual object locations), and assumes that no more than some maximum number of 

13 the virtual objects can be or are desired to be assigned to a given virtual object location 37. This 

14 amount is denoted as [MAX VIRTUAL OBJECTS] . 

15 Figure 15 presents an embodiment of a process used by the virtual object placement 

16 engine 1307 to execute the functions listed in blocks 2368, 2370, 2372, and 2374 depicted in 

17 Figure 14. The process begins with the start ellipse, 2318. In block 2320, the virtual object 

18 placement engine 1307 determines the virtual object 38 best suited for each virtual object 

19 location 37 for all target categories. In block 2322, the virtual object placement engine 1307 

20 determines the best virtual object/target category combination for each virtual object location 37. 

2 1 In block 2324, the virtual obj ect placement engine 1307 compares virtual obj ect/target category 

22 combinations for all virtual object locations. In block 2326, the virtual object placement engine 

23 1307, for a virtual object location 37 and target category, determines the best virtual objects to 

24 associate with the virtual object location 37. In block 2328, the virtual object placement engine 

25 1307 repeats block 2326 for each target category. In block 2330, the virtual object placement 

26 engine 1307 determines the target category that yields the optimum placement of virtual obj ects 

27 for a virtual object location 37. In block 2332, the virtual object placement engine 1307 repeats 

28 blocks 2326, 2328, and 2330 for all virtual object locations. In block 2334, the virtual object 

29 placement engine 1307 determines the best combination of multiple virtual objects for each 

30 virtual object location 37. In block 2336, for the remaining virtual object locations, the virtual 
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1 object placement engine 1 307 assigns the best matching virtual object 38. The process ends with 

2 block 2338. 

3 A further embodiment of a virtual objects targeting algorithm presented in Figure 15 

4 will be described with reference to the example values shown in Tables A-E. Various other 

5 prioritizing or ranking schemes may be used as described later. 

6 Step 1: In block 2320 in Figure 15, the virtual object placement engine 1307, for a 

7 virtual object location 37, determines the virtual objects with the highest overall ranking if that 

8 virtual obj ect 3 8 were the only virtual obj ect 3 8 to be placed in a virtual obj ect location 3 7 in the 

9 content 36. This step compares the data in Tables D and E. Figure 16 and the description that 

1 0 follows below present a more detailed embodiment of several of the blocks presented in Figure 

11 1 5 . In step 1 a 5 as an embodiment of block 242 1 in Figure 1 6, the virtual obj ect placement engine 



12 1 307 selects the first virtual obj ect location 3 7 and as an embodiment of block 242 1 in Figure 1 6, 

13 selects the first virtual object 38 to be analyzed. As Step lb, for that virtual object selected in 

14 Step la, the virtual object placement engine 1307 selects the first category, as an embodiment of 

15 block 2423 in Figure 16. Then, the virtual object placement engine 1307 multiplies the virtual 

16 object's Group Ranking Percentage by the virtual object location's Group Breakdown Percentage 

17 for each group as an embodiment of block 2424 in Figure 16 and sums the result, as an 

18 embodiment of block 2425 in Figure 16. As Step lc, the virtual object placement engine 1307 

19 repeats Step lb for the next target category, as an embodiment of block 2426 in Figure 16. As 

20 Step Id, the virtual object placement engine 1307 repeats steps lb and lc for each virtual object 

21 38, as an embodiment of block 2427 in Figure 16. As Step le, for the virtual object location 37 

22 under consideration, the virtual object placement engine 1307 selects the virtual object/target 

23 category that yields the highest summed value, as an embodiment of block 2428 in Figure 16. 

24 Then, for Step If, the virtual object placement engine 1307 repeats Steps lb-le for all virtual 



25 object locations, as an embodiment of block 2429 in Figure 16. 

26 For example, using virtual object location 1, virtual object 1 : 

27 target category 1: 50*25 + 50*25 + 0*25 + 0*25 = 25% 

28 target category 2: 30*30 + 10*10 + 20*20 + 40*40= 30% 

29 target category 3: 0*10 + 0*10 + 0*20 + 0*20 0*20 + 0*20- 0% 
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1 The cross-multiplied result then shows ameasure of effectiveness for each virtual object 

2 38 if displayed in the corresponding virtual object location 37. Table F below presents the 

3 results of Step 1 above for virtual object location 1 
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Table F 



Virtual 
object 
location 
/ Virtual 
object 


Target 
Category 


Group 
1 


Group 
2 


Group 

3 


Group 
4 


Group 
5 


Group 
6 


Sum- 
mation 


1/1 


1 


12.5 


12.5 


0 


0 


0 


0 


25 




2 


9 


1 


4 


16 


0 


0 


30 




3 


0 


0 


0 


0 


0 


0 


0 


1/2 


1 


0 


0 


12.5 


12.5 


0 


0 


25 




2 


0 


0 


0 


0 


0 


0 


0 




3 


0 


0 


0 


0 


0 


0 


0 


1/3 


1 


0 


0 


0 


0 


0 


0 


0 




2 


7.5 


2.5 


5 


10 


0 


0 


25 




3 


0 


0 


0 


0 


0 


0 


0 


1/4 


1 


12.5 


0 


12.5 


0 


0 


0 


25 




2 


0 


4 


8 


8 


0 


0 


20 




3 


1 


3 


2 


6 


2 


2 


16 


1/5 


1 


10 


5 


5 


5 


0 


0 


25 




2 


3 


3 


6 


12 


0 


0 


24 




3 


3 


3 


6 


1 


1 


0 


14 


1/6 


1 


0 


0 


0 


0 


0 


0 


0 




2 


0 


0 


0 


0 


0 


0 


0 




3 


1 


1 


2 


2 


6 


6 


18 


1/7 


1 


5 


5 


10 


5 


0 


0 


25 




2 


7.5 


2.5 


5 


10 


0 


0 


25 




3 


0 


3 


4 


6 


0 


4 


17 


1/8 


1 


7.5 


10 


0 


7.5 


0 


0 


25 




2 


9 


3 


2 


12 


0 


0 


26 




3 


2 


0 


4 


4 


4 


4 


18 
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1 Step 2: Returning to Figure 15, for each virtual object location 37, the virtual object 

2 placement engine 1 3 07, in block 2322, determines the virtual obj ect/target category combination 

3 that results in the highest overall ranking. In one embodiment the virtual object placement 

4 engine 1307, lists the virtual object locations, the overall ranking, the corresponding virtual 

5 object 38, and the corresponding target category. In case of a tie, the virtual object placement 

6 engine 1307 selects any virtual object 38 with the overall highest ranking. Table G shows the 

7 results. Thus, from Table G, virtual object 4, a virtual object 38 displayed within virtual object 

8 location 4 yields a measure of effectiveness of 50 (highest) and virtual object 8 along within 

9 virtual object location 5 yields a measure of effectiveness of 28. 
10 Table G 



Virtual Object 
Location 


Highest Overall 
Ranking 


Corresponding 
Virtual Object 


Corresponding 
Target Category 


Virtual object location 
1 


30 


Virtual Object 1 


2 


Virtual object location 
2 


35 


Virtual Object 2 


1 


Virtual object location 
3 


35 


Virtual Object 1 


1 


Virtual object location 
4 


50 


Virtual Object 4 


1 


Virtual object location 
5 


28 


Virtual Object 8 


2 



11 Step 3: In one embodiment of block 2324 in Figure 15, the virtual object placement 

12 engine 1 307 orders the resulting list of virtual object locations from Step 2 from lowest overall 

13 ranking to highest overall ranking to compare virtual object/target category combinations for 

14 virtual object locations. Table H shows the results. 
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Table H 



Virtual Object 
Location 


Overall Ranking 


Corresponding 
Virtual Object 


Corresponding 
Target Category 


Virtual object location 
5 


28 


Virtual Object 8 


2 


Virtual object location 
1 


30 


Virtual Object 1 


2 


Virtual object location 
2 


35 


Virtual Object 2 


1 


Virtual object location 
3 


35 


Virtual Object 1 


1 


Virtual object location 
4 


50 


Virtual Object 4 


1 



Step 4: In one embodiment of block 2326 in Figure 15, the virtual object placement 
engine 1307 uses the process shown in Figure 17 to determine the best virtual objects to 
associate with a virtual object location 37. The block begins with ellipse 2440. In block 2441 in 
Figure 17, the virtual object placement engine 1307 selects the virtual object location 37 from 
Step 3 resulting in the lowest overall ranking. As Step 4a ? for the selected virtual object location 
37, the virtual object placement engine 1307 selects the first target category, as an embodiment 
of block 2442 in Figure 17. As Step 4b, the virtual object placement engine 1307 assembles a 
table showing the product of each virtual object Group Ranking Percentage and virtual object 
location Group Breakdown Percentage combination. Table I below provides an example for 
virtual object location 5 and target category 1. 



Table I 



Virtual Object 
Location /Virtual 
Object 


Target 
Category 


Group 
1 


Group 
2 


Group 
3 


Group 
4 


Sum- 
mation 


5/1 


1 


10 


15 


0 


0 


25 


5/2 


1 


0 


0 


15 


10 


25 
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Virtual Object 
Location / Virtual 
Object 


Target 
Category 


Group 
1 


Group 
2 


Group 
3 


Group 
4 


Sum- 
mation 


5/3 




0 


n 








5/4 




1 0 


o 




A 
U 


Id 






0 

o 


o 


0 


/t 
4 


0/1 


5/6 




0 


0 


0 


o 


o 


5/7 




4 


6 


12 


4 


26 


5/8 




6 


12 


0 


6 


24 



1 As Step 4c, as an embodiment of block 2443 in Figure 17, the virtual object placement 

2 engine 1307 finds the product that is the highest. In case of a tie, the virtual object placement 

3 engine 1 3 07 selects the product that corresponds to the highest summation value for that virtual 

4 object location / virtual object combination. In case a tie still persists, the virtual object 

5 placement engine 1307 selects any of the cells with an equivalent value. Table J below shows 

6 the previous example continued where group 2 for virtual object location / virtual object 

7 combination 5/1 is selected. 

8 Table J 



Virtual Object 
Location / 
Virtual 
Object 


Target 
Category 


Group 
1 


Group 
2 


Group 
3 


Group 
4 


Sum- 
mation 


5/1 


1 


10 


*15* 


0 


0 


25 


5/2 


1 


0 


0 


15 


10 


25 


5/3 


1 


0 


0 


0 


0 


0 


5/4 


1 


10 


0 


15 


0 


25 


5/5 


1 


8 


6 


6 


4 


24 


5/6 


1 


0 


0 


0 


0 


0 • 
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Virtual Object 


Target 


Group 


Group 


Group 


Group 


Sum- 


Location / 


Category 


1 


2 


3 


4 


mation 


V 11 llldl 














Object 














5/7 


1 


4 


6 


12 


4 


26 


5/8 


1 


6 


12 


0 


6 


24 



1 Step 5: As an embodiment of block 2444 in Figure 17, the virtual object placement 

2 engine 1307 finds the product that is next highest (or the same value as in Step 4), but that is 

3 associated with a group not yet selected. Again, in case of a tie, the virtual object placement 

4 engine 1 307 selects the product that corresponds to the highest summation value for that virtual 

5 object location / virtual object combination. In case a tie still persists, the virtual object 

6 placement engine 1307 selects any of the cells with an equivalent value. Table K below shows 

7 the previous example continued. 

8 Table K 



Virtual Object 
Location /Virtual 
Object 


Target 
Category 


Group 1 


Group 2 


Group 3 


Group 4 


5/1 




*10* 


*15* 


0 


0 


5/2 




0 


0 


*15* 


*10* 


5/3 




0 


0 


0 


0 


5/4 




10 


0 


15 


0 


5/5 




8 


6 


6 


4 


5/6 




0 


0 


0 


0 


5/7 




4 


6 


12 


4 


5/8 




6 


12 


0 


6 



9 Step 6: As an embodiment of block 2446 in Figure 17, the virtual object placement 

10 engine 1307 repeats Step 5 until a product has been selected for all groups. Table L below 

1 1 continues the example. 
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1 



Table L 



Virtual Object 
Location / 

\/ii* , l"HQi flriiiif*^" 
V 11 IU<11 v/UjCll 


Target 
Category 


Group 1 


Group 2 


Group 3 


Group 4 


5/1 


1 


no* 


*15* 


0 


0 


5/2 


1 


0 


0 


*15* 


*10* 


5/3 


1 


0 


0 


0 


0 


5/4 


1 


10 


0 


15 


0 


5/5 


1 


8 


6 


6 


4 


5/6 


1 


0 


0 


0 


0 


5/7 


1 


4 


6 


12 


4 


5/8 


1 


6 


12 


0 


6 



2 Step 7: As an embodiment of block 2448 in Figure 17, for all virtual objects with 

3 products cells selected in Step 6, the virtual object placement engine 1 307 calculates the summed 

4 products of those selected cells for each virtual object 38. Table M below shows the results. 

5 Table M 



Virtual Object 
Location / 
Virtual Object 


Target 
Category 


Group 1 


Group 2 


Group 3 


Group 4 


Sum- 
mation 


5/1 


1 


*10* 


*15* 


0 


0 


25 


5/2 


1 


0 


0 


*15* 


*10* 


25 


5/3 


1 


0 


0 


0 


0 


0 


5/4 


1 


10 


0 


15 


0 


0 


5/5 


1 


8 


6 


6 


4 


0 


5/6 


1 


0 


0 


0 


0 


0 


5/7 


1 


4 


6 


12 


4 


0 
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Virtual Object 


Target 


Group 1 


Group 2 


Group 3 


Group 4 


Sum- 


Location / 


Category 










mation 


Virtual Object 














5/8 


1 


6 


12 


0 


6 


0 



1 Step 8: As an embodiment of block 2450 in Figure 17, the virtual object placement 

2 engine 1 307 orders the virtual objects in Step 7 from highest summed value to lowest. In case of 

3 equal summed values, the virtual object placement engine 1307 arbitrarily orders those virtual 

4 objects with the same summed value. Table N presents the example results. 

5 Table N 



Virtual Object 


Target 


Group 1 


Group 2 


Group 3 


Group 


Sum- 


Location / 


Category 








4 


mation 


Virtual Object 














5/1 


1 


10 


15 


0 


0 


25 


5/2 


1 


1 


0 


15 


10 


25 



6 Step 9: As Step 9a, if the number of virtual objects selected in Step 8 exceeds 

7 [MAX_VIRTUAL OBJECTS], the virtual object placement engine 1307 selects the first 

8 [MAX VIRTUAL OBJECTS] virtual objects with the summed value as an embodiment of block 

9 2452 in Figure 1 7. For example, if it is desired to assign at most two virtual objects to a virtual 

10 object location 37, the virtual object placement engine 1307 selects the two virtual objects with 

11 the highest virtual object Group Ranking Percentage and virtual object location Group 

12 Breakdown Percentage products. Next, as Step 9b, for the unselected virtual objects, the virtual 

13 object placement engine 1 307 determines those groups that were associated with these omitted 

14 virtual objects, as an embodiment of block 2454 in Figure 17. 

15 Step 10: As an embodiment of block 2456 in Figure 17, for the virtual objects 

16 associated with the groups determined in Step 9b, the virtual object placement engine 1307 

17 selects the product within that group that is the highest for the [MAX_VIRTUAL OBJECT] 

18 selected virtual objects from Step 9a. The virtual object placement engine 1 307 recalculates the 

19 summed products of those selected groups cells for each of the virtual objects. Table O below 

20 provides a new example, assuming [MAX J/IRTUAL OBJECTS] - 2; therefore, groups 5 and 6, 



44 



5221 



1 which are associated with virtual object 6, may be reallocated to virtual objects 7 & 5, 

2 respectively. 

3 Table O 

4 Result before Step 10 is shown below: 



Virtual 
object 
location / 
Virtual 
object 


Target 
Category 


Group 
1 


Grou 
p2 


Group 
3 


Grou 

P 
4 


Group 
5 


Group 
6 


Sum- 
mation 


5/7 


3 


0 




2 




0 


2 


18 


5/5 


3 




9 




1.5 


0.5 


0 


6 


5/6 


3 


1 


3 


1 


3 


*3* 


*3* 


6 



5 

5 Result after Step 10 is shown below: 



Virtual 
object 
location / 
Virtual 
object 


Target 
Category 


Group 
1 


Grou 
p2 


Group 
3 


Group 
4 


Group 
5 


Group 
6 


Sum- 
mation 


5/7 


3 


0 




2 




0 


*2* 


20 


5/5 


3 


*3* 


9 


*3* 


1.5 


*0.5* 


0 


6.5 


5/6 


3 


1 


3 


1 


3 


3 


3 


0 



6 Step 11: As an embodiment of block 2458 in Figure 17, the virtual object placement 

7 engine 1307 calculates the total summed product value for all virtual objects selected inStep 10. 

8 From Table P, this value is 26.5. The resultant groups selected for each virtual object 38 will 

9 serve as the group assignments if this virtual obj ect location / target category ultimately results in 
10 the best match, as determined in the remaining steps of the algorithm. 
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1 



Table P 



Virtual 
object 
location / 
Virtual 
object 


Target 
Category 


Grou 

pi 


Group 
2 


Group 
3 


Group 
4 


Group 
5 


Group 
6 


Sum- 
mation 


5/7 


3 


0 




2 


*g* 


0 


*2* 


20 


5/5 


3 




9 


* 3 * 


1.5 


0.5 


0 


6.5 


Total summed product values 


26.5 



2 Step 12: The virtual object placement engine 1307 repeats steps 4-11 above for the 

3 same selected virtual object location 37 of Step 4 using the remaining target categories, as an 

4 embodiment of block 2328 in Figure 1 5 . The Table Q example below provides the output results 

5 for each of the three example target categories. 
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Table Q 



2 
3 
4 
5 
6 



Virtual 
object 
location / 
Virtual 
object 


Target 
Category 


Group 
1 


Group 
2 


Group 

3 


Group 
4 


Sum-ma tion 


5/1 


1 


*10* 


*15* 


0 


0 


25 


5/2 


1 


0 


0 


*15* 


*10* 


25 


Total summed product values 


50 


Virtual 
object 
location / 
Virtual 
object 


Target 
Category 


Group 
1 


Grou 
P 2 


Group 
3 


Group 
4 


Group 
5 


Group 
6 


Sum- 
mation 


5/1 


2 




3 


2 




0 


0 


21 


5/4 


2 


0 


*12* 


*4* 


6 


0 


0 


16 


Total summed product values 


37 


Virtual 

object 

location 

/Virtual 

object 


Target 
Category 


Group 
1 


Group 
2 


Group 
3 


Group 
4 


Group 
5 


Group 
6 


Sum- 
mation 


5/7 


3 


0 




2 


*o,* 


0 


*2* 


20 


5/5 


3 


*3* 


9 




1.5 


*0.5* 


0 


6.5 


Total summed product values 


26.5 



Step 13: As an embodiment of block 2330 in Figure 15, the virtual object placement 
engine 1307 selects the target category that yields the highest total summed product amount. 
The virtual object placement engine 1307 assigns this as the Maximum Rank for that virtual 
object location 37. In the case above, the virtual object placement engine 1307 would assign 
target category 1, with a value of 50 that is selected. 
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1 Step 14: As an embodiment of block 2332 in Figure 15, the virtual object placement 

2 engine 1 3 07 repeats Steps 4- 1 3 for the virtual obj ect location 3 7 selected in Step 4 with the next 

3 lowest overall ranking, computing the Maximum Rank for each virtual object location 37. 

4 Step 15: As an embodiment of block 2334 in Figure 15, the virtual object placement 



5 engine 1307 uses the available [MAXJVIRTUAL OBJECTS] virtual objects for the virtual 

6 object locations up to the maximum number of [TOTAL_VIRTUAL OBJECTS] that yield the 

7 largest Maximum Rank. The virtual object placement engine 1307 makes use of the relevant 

8 target category determined in Step 1 3 , with virtual obj ects as determined in Step 1 0, with group 

9 assignments as determined in Step 1 1 . 



10 Step 16: As an embodiment of block 2336 in Figure 15, for all other virtual object 

1 1 locations, the virtual obj ect placement engine 1 3 07 assigns the single virtual obj ects that yielded 

12 the highest Overall Ranking as determined in Step 2. 

1 3 The above algorithm performed by the virtual obj ect placement engine 1 307 is meant to 

14 be illustrative and not limiting. Other algorithms are possible for assigning targeted virtual 

15 objects to groups of reception sites or to individual reception sites. Other targeted virtual object 

16 routines can also be used by the virtual object placement engine 1307. 

17 The above algorithm can be simplified in the case where virtual objects are being 

1 8 selected to be delivered with the content 36 to be received by a single subscriber or reception site 

19 30. In this case, prior to initiating the steps in the algorithm, the virtual object location Group 

20 Breakdown Percentages table may be modified to display a group breakdown percentage of 0 for 

21 all groups that the subscriber does not belong to for each target category. 

22 An alternate virtual object targeting routine 1 374 is described in U.S. Patent 5,600,364, 

23 to Hendricks, John S, entitled NETWORK CONTROLLER FOR CABLE TELEVISION 

24 DELIVERY SYSTEM, which is hereby incorporated by reference. In this alternative, software 

25 in the virtual obj ect targeting system 1 220 generates packages of virtual obj ects geared towards 

26 particular subscribers and makes use of a subscriber's demographic information and viewing 

27 habits to determine those virtual objects that are of most interest to that particular subscriber. 

28 The routine 1374 then outputs packages of virtual objects targeted towards each subscriber or 

29 group of subscribers. 

30 Figure 18 shows the seven primary functions of an alternate virtual object targeting 

31 routine 1374. The function of the routine 1374 is to target virtual objects for reception sites 
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1 based on historical programs watched data and other data that is available at the TVOMS 300. 

2 In the discussion that follows, the alternate virtual object targeting routine 1374 is described as 

3 executed at the TVOMS 300. 

4 The process may be initiated as shown at initiation ellipse 1420. In the first subroutine, 



5 identified at block 1422, the virtual object targeting system 1220 determines the programs 

6 watched matrices stored in the subscriber information database 1210. The determine programs 

7 watched matrices subroutine 1422 uses a unique reception site ID to access a specific matrix for 

8 one reception site. These matrices are maintained and updated by periodic collections by the 



9 operations center 1 0 of accumulated information from the reception sites. 

10 In the second subroutine, shown at block 1 424, the virtual obj ect targeting system 1 220 

1 1 develops other matrices based on other available information. To develop other matrices based 

12 on other available information subroutine 1424 is an optional subroutine not required for the 

13 functioning of the system. For groups of reception sites or for each individual reception site, 

14 matrices may be developed based on the demographic information, billing information, pricing 

15 information, age information and other information that may be stored in the subscriber 

1 6 information database 1210. 

IV In the process matrices through correlation algorithms subroutine, block 1426, the 

18 virtual object targeting system 1220 processes all matrices through a set of correlation 

19 algorithms. In particular, the virtual object targeting system 1220 takes matrices developed in 

20 the first two subroutines 1422 and 1424 and processes the matrices until reaching a final matrix. 

21 Figure 19 shows an embodiment of the matrices processing subroutine 1426 that is 



22 called by the virtual objects targeting sequence 1374 shown in Figure 18. As shown in Figure 

23 1 9, the virtual obj ect targeting system 1 220 initiates the matrices processing subroutine 1 426 at 

24 initiation ellipse 1 427 and then accesses or queries, at block 1 420, the programs watched file and 

25 gathers information regarding either an individual subscriber or a group of subscribers. The 

26 virtual obj ect targeting system 1 220 can gather the programs watched information in this way for 

27 individual subscribers or a group of subscribers. 

28 Once the programs watched information has been gathered in the database, the virtual 

29 obj ect targeting system 1 220 selects and groups, at block 1 430, programs watched categories and 

30 time periods. The software initially takes each program category (e.g., sports, news, mysteries, 

3 1 etc) and determines the number of programs watched for a given time. The periods may be set 
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1 to any length of time, including, for example, one, two, three or four weeks. The virtual object 

2 targeting system 1220 will loop through such a counting process for each group and period and 

3 then proceed to build a programs watched matrix, at block 1432, based on the program 

4 categories and periods. Essentially, all programs watched in a particular category and time 

5 period will be entered into the programs watched matrix. Once the matrix has been built, the 

6 virtual object targeting system 1220, using matrices processing subroutine 1426, will process the 

7 matrix for a given subscriber or group of subscribers through the correlation algorithms. 

8 A number of correlation algorithms may be used to weight each selected program 

9 category. For example, as shown at block 1434, the virtual object targeting system 1220 may 

10 use a sum of squares algorithm to determine the weighting. Once weighted, the weighted 

11 categories will be correlated by the virtual object targeting system 1220 at block 1436, with 
... 12 various virtual objects stored in the available virtual objects database 1265. The virtual object 
IV 1 3 targeting system 1 220 then selects a set of the most heavily weighted virtual objects for inclusion 
„, ; 14 within the content 36 to be delivered to individual subscribers or groups of subscribers. Having 
^.15 determined the weightings of each group and prioritizing the groups accordingly, the virtual 

- 16 ' object targeting system 1220 returns, block 1438, to the virtual objects targeting sequence 1374 
" 17 of Figure 18. 

V; 18 Referring back to Figure 1 8, in the fourth subroutine, as represented at block 1428, the 

- 19 virtual object targeting system 1220 uses the final matrix developed by the correlation and 
* 20 weighing algorithm described above, to select a grouping (or selective filter) for each reception 
~ 21 site 30. The final groupings of virtual objects that may be sent to the reception sites or group of 

22 reception sites may use a subroutine as diagramed in Figure 20. 

23 The fourth subroutine 1428, depicted in Figure 20, is called or initiated by the virtual 

24 objects targeting sequence 1374 of Figure 18 in order to determine the final groupings. In the 

25 subroutine shown at block 1444, the virtual object targeting system 1220 selects a set of virtual 

26 objects that will be used in the chosen groupings. This selection process may involve virtual 

27 objects from various virtual objects categories. Each virtual object 38 may subsequently be 

28 assigned a number of times that it will be shown in a given segment of content 36. The 

29 frequency of display may be based on various factors, including the number of requests and cost 

3 0 paid by the respective advertisers to have the virtual obj ects displayed, as shown in block 1 446 . 

31 Such factors may be used by the virtual object targeting system 1220 in the next step of the 
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1 subroutine, at block 1448, at which the virtual object targeting system 1220 assigns a weighting 

2 to specific virtual objects in each virtual objects category. These weightings are used to 

3 prioritize the virtual objects that will be sent to individual reception sites or group of reception 

4 sites. 

5 Once the virtual objects have been weighted, the virtual object targeting system 1220 

6 executes a correlation algorithm, at block 1450, using selected criteria (i.e., the various factors 

7 used to weight the virtual objects) as well as the output of each programs watched matrix. Any 

8 number of correlation algorithms and weighting algorithms may be used, including the sum of 

9 squares weighting algorithm described above. 

10 The results from the correlation algorithm subsequently determine the virtual objects 

1 1 and program content 36 that is sent to the virtual object targeting system 1220 for distribution. 

12 Once the virtual object targeting system 1220 at the fourth subroutine 1428 completes these 

13 steps, the subscriber information database 1210 updates the subscriber record based on the 

14 virtual objects that are sent, as shown at block 1454. The database update allows the advertisers 

15 to track the costs and frequency of the virtual objects targeted to specific reception sites or 

1 6 groups of reception sites. Following the updates, the virtual object targeting system 1220 returns 

17 to the virtual objects targeting sequence shown in Figure 18, block 1456. 

18 Referring to Figure 21, reception site groupings (1 through 5) 1460 are shown. The 

19 number of reception site groupings available may be determined by the bandwidth available to 

20 transmit virtual objects along with content 36. The available bandwidth or resources provided by 

2 1 the delivery network 1 1 may limit the number of virtual obj ects that are available to distribute to 

22 the reception site 30. 

23 Referring back to Figure 18, the virtual object targeting system 1220 at the fifth 

24 subroutine, represented at block 1 466, prepares reception site group information for transmission 

25 to the reception sites along with the requested content 36. 

26 In the sixth subroutine, block 1468, the virtual object targeting system 1220 selects the 

27 targeted virtual objects. The sixth subroutine 1468 is the last decision making process in 

28 displaying a targeted virtual obj ects for a subscriber. As shown in block 1 469, the reception site 

29 30 then displays the targeted virtual objects with the content 36. 

30 As noted above, targeted advertising can be based on viewing a specific program or a 

3 1 category of programming content 36. In an embodiment, the reception site 30 performs this last 
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1 step by correlating (or matching) the program being watched by the subscriber with the reception 

2 site group information that has been previously transmitted by the TVOMS 300. Figure 21 

3 shows an exemplary table matching reception site groups 1460 and program categories 1470 

4 with specific virtual obj ects. The virtual obj ects are shown in Figure 22 at 1 474 and are assigned 

5 Roman numerals I through X, for example. The number of reception site groupings and virtual 

6 objects can vary. Figure 22 shows a division of available bandwidth to carry ten virtual objects. 

7 In this example, the virtual objects 1474 are numbered 1101-1110. 

8 The TVOMS 300 will transmit group information to a reception site 30 shown as row 

9 names 1460 on Figure 21 . The TVOMS 300 will also transmit data that informs the reception 

10 site 30 which of the multiple virtual objects 1474 is assigned to a program category shown as 

11 columns 1470 on Figure 21. Each reception site 30 only requires the data related to that 

12 reception site's assigned group (or row). For example, in Figure 21, the reception site 30 in 

13 group 1 (row 1) is provided with data on the virtual objects which are assigned for sports 

14 program as I, children's program as IV and mystery category program as III. In this manner, 

15 each reception site 30 is only required to store information related to its own grouping. 

16 Therefore, a reception site 30 that is in group 1 only needs to store the information related to 

17 group 1 that is found in row 1 of Figure 21. 

1 8 Figure 23 shows a software program flow 1 490 that is an alternative to the virtual object 

19 targeting system 1220 targeting routine 1374, depicted in Figure 18. The alternative routine 

20 1490 allows each reception site 30 to be individually targeted with specific virtual objects. 

21 Preferably, it is initiated automatically, as shown at block 1492, by the TVOMS 300 upon receipt 

22 of a program request from a reception site, for example, for a pay per view program. Thus, once 

23 the TVOMS 300 receives program request information from a reception site, the TVOMS 300 

24 begins the process of selecting a package of virtual objects that may be based on, among other 

25 things, that subscriber's demographic information and viewing history. 

26 Upon receipt of a program request from a reception site, the virtual object targeting 

27 system 1220 reads the reception site identifier, as shown at block 1494, and the program 

28 requested. The subscriber data collection engine 1202 writes information on the program 

29 requested to the subscriber information database 1210, updating the subscriber record that 

30 contains listings of all programs requested within the past week, month or year. 
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1 With continued reference to Figure 23, the virtual object targeting system 1220 then 

2 calls a subroutine that sorts the programs requested by program category, block 1498. In turn, 

3 the program categories are sorted, as shown at block 1 500, based on the number of times that 

4 program appearing in each particular category is requested. In so doing, virtual object targeting 

5 system 1 220, using the sorting subroutine as shown at block 1 500, determines and ranks those 

6 programs and program categories that are most frequently viewed at that reception site. 

7 All rankings of programs and program categories for that reception site 30 are written to 

8 the subscriber information database 1210, as shown at block 1502. 

9 Next, the virtual obj ect targeting system 1 220 calls a subroutine, shown at block 1 504, 

10 that correlates the updated subscriber record with the available virtual objects database 1265. By 

1 1 correlating these two with one another, the subroutine assigns or correlates various categories of 

12 virtual objects to each ranking of programs and program categories. The categories of virtual 

13 objects that may be so assigned are found in the available virtual objects database 1265 and may 

14 include: (1) Household Goods/Products, (2) Home Improvement and Maintenance, (3) Personal 

15 Hygiene, (4) Entertainment Items and Events, (5) Sporting Goods and Events, (6) Motor 

16 Vehicles and Related Products, (7) Foodstuffs and Beverages, and (8) Miscellaneous, for 

17 example. Where, for example, the subscriber has watched a sporting program, the Sporting 

18 Goods and Events, Home Improvement and Maintenance categories may be assigned to that 

19 particular sporting event/ program and Sports program category, for example. 

20 Once the programs and program categories are correlated with the virtual objects 

21 categories in the available virtual objects database 1265, the virtual object targeting system 1220 

22 calls a sorting subroutine 1 506 that ranks the correlated virtual obj ects categories based on other 

23 information in the database files. In one embodiment, this ranking is primarily based on data in 

24 the updated subscriber information database 1210, as shown at block 1506. By using data on the 

25 subscriber's past program selections and demographic information, the virtual object targeting 

26 system 1220 ranks the correlated categories of virtual obj ects according to those likely to be of 

27 most interest to that subscriber. 

28 After the virtual object categories have been sorted and ranked, the virtual object 

29 targeting system 1220 selects the top three virtual objects categories as the targeted categories 

30 for a given program and subscriber, block 1508. Individual virtual obj ects are then chosen from 

31 the available virtual objects database 1265, with all selections made from the targeted categories, 
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1 at block 1510. The virtual objects that are selected are written to the subscriber information 

2 database 1210 and to the content and virtual object packager 30, from where packages can be 

3 generated, at block 1512, for ultimate delivery to the reception site. 

4 Figure 24 depicts the obj ect delivery center 15. The obj ect delivery center 1 5 receives 

5 content 36, virtual objects, retrieval plans, and other information from the operations center 10 

6 that is to be transmitted to reception sites. The communication processor 16 in the object 

7 delivery center 1 5 may determine the delivery network and communications methods appropriate 

8 for each item to be delivered, may combine items to be delivered to common destinations, may 

9 format the items for delivery, and provide the formatted items to the processing router 17. The 

10 processing router 17 may then route each item to the appropriate modular connector 700, for 

11 example modular connector 700*, modular connector 700", or modular connector 700'", 

12 depending on the required delivery network 1 1 and communication method. 

13 A number of embodiments of delivery networks 1 1 , 12, and 14 are presented below. 

14 The embodiments presented below may use the object delivery center 15, which inserts the 

1 5 virtual obj ects into the signal for delivery over the delivery network 1 1 or 1 2. The embodiments 

1 6 presented below use a modular connector 700 in the reception site 30, that receives the delivered 

17 signal with virtual objects, extracts the virtual objects, and provides the virtual objects to the 

18 storage management processor 710. The modular connector 700 supports the receive 

1 9 functionality for each unique delivery network communication method embodiment. 

20 Figure 25 presents embodiments associated with the delivery of virtual objects over a 

21 coaxial or fiber cable system 2701 to a reception site 30. Virtual objects are provided to the 

22 delivery network 1 1 by the object delivery center 15 or directly by the operations center 10. 

23 Alternatively, content 36 and virtual objects may be provided to the reception site 30 from the 

24 object delivery center 15 or from the local insertion center 20 using delivery network 12. The 

25 signal is delivered over the cable system 2701 delivery network. The signal may provide for the 

26 delivery of virtual objects, content 36 containing virtual object locations, and reception site 

27 configuration and control information. The signal may also provide for virtual object viewing 

28 data and interactive virtual object requests from the reception site 30 to the local data collection 

29 center 40, to the central data collection center 50, or to the interactive object service center 60 

30 using delivery network 14 or the signal may be a means to provide access to the Internet or other 

3 1 public network through which virtual objects or content 36 are delivered (not shown). The cable 
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1 



system 2701 may be a coaxial cable network, totally fiber network, hybrid fiber coax network, 

2 fiber to the curb network, or any other cable distribution technology. The signal over the cable 

3 system may be generated by a cable modem, in which an external cable modem 2702 is used to 

4 receive the signal and provide the embedded virtual obj ects to the modular connector 700 in the 

5 reception site 30 for processing. Alternatively, the reception site 30 may contain an internal 

6 cable modem 2705, which receives the signal and provides the virtual objects to the modular 

7 connector 700 for processing. 

8 In another embodiment, the signal delivered over the cable system is a video signal. In 

9 one embodiment, the video signal is an analog video signal. In another embodiment, the video 

10 signal is a digital video signal. The reception site 30 may contain an internal cable 

1 1 receiver/tuner/demodulator 2706 to process the signal, and provide the embedded virtual objects 

12 to the modular connector 700. A set top terminal 2703, or other device capable of receiving a 

1 3 cable video signal, such as a cable ready TV, or PC with cable tuner (not shown), may process 

14 the video signal and deliver the video signal to the connector 700 in the reception site 30, which 

15 extracts the embedded virtual objects. Alternately, the set top terminal 2703, or other such 

1 6 device, may extract the embedded virtual obj ects from the video signal and provide the virtual 

1 7 obj ects to the modular connector 700 in the reception site 30 
In another embodiment, virtual objects may be embedded within the audio signal, 

1 9 requiring an appropriate audio-capable modular connector 700 in the reception site 30 to extract 

20 the virtual obj ects from the audio signal. In one embodiment, the audio signal is an analog audio 

21 signal. In another embodiment, the audio signal is a digital audio signal. 

22 In yet another embodiment, the signal is a spread spectrum signal containing a digital 

23 data stream, requiring an appropriate spread spectrum receiver and modular connector 700 in the 

24 reception site 30 to extract the virtual obj ects. In this embodiment, the spread spectrum signal is 

25 transmitted in the same bandwidth as the video or audio signal, but below the noise level. 

26 Figure 26 presents embodiments associated with the delivery of virtual objects over a 

27 wireless broadcast system 2801 to a reception site 30. Virtual objects are provided to the 

28 delivery network 1 1 by the object delivery center 15 or directly by the operations center 10. 

29 Alternatively, content 36 and virtual objects may be provided to the reception site 30 from the 

30 object delivery center 1 5 or from the local insertion center 20 using delivery network 12. The 

3 1 signal is delivered over the wireless broadcast system 2801 delivery network. The signal may 
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1 provide for the delivery of virtual objects, content 36 containing virtual object locations, and 

2 reception site configuration and control information. The signal may also provide for virtual 

3 obj ect viewing data and interactive virtual obj ect requests from the reception site 30 to the local 

4 data collection center 40, to the central data collection center 50, or to the interactive object 

5 service center 60 using delivery network 1 4 or the signal may be a means to provide access to 

6 the Internet or other public network through which virtual objects or content 36 are delivered. 

7 The wireless broadcast system may be a microwave multipoint delivery system (MMDS), local 

8 multipoint distribution system (LMDS), Instructional Television Fixed Service (ITFS) system, or 

9 any other wireless data, video, or telephony broadcast system, including point-to-point and 

10 point-to-multipoint microwave broadcast systems like those provided by Teligent, Winstar 

11 digital wireless network, and ATT's wireless system. The signal over the wireless broadcast 

12 system may be generated by a wireless modem, in which an external wireless modem 2802 is 

1 3 used to receive the signal and provide the embedded virtual obj ects to the modular connector 700 

14 in the reception site 30 for processing. Alternatively, the reception site 30 may contain an 

1 5 internal wireless modem 2805 , which receives the signal and provides the virtual obj ects to the 

1 6 modular connector 700 in the reception site 30 for processing. 

17 In another embodiment, the signal delivered over the wireless broadcast system is a 

18 video signal. In one embodiment, the video signal is an analog video signal. In another 

19 embodiment, the video signal is a digital video signal. The reception site 30 may contain an 

20 internal wireless receiver/tuner/demodulator 2806 to process the signal, and provide the 

21 embedded virtual objects to the modular connector 700. A wireless set-top terminal 2803, or 

22 other device capable of receiving a wireless video signal, such as a TV, or PC with a wireless 

23 receiver and tuner, may process the video signal and deliver the video signal to the modular 

24 connector 700 in the reception site 30, which extracts the embedded virtual objects. Alternately, 

25 the set top terminal 2803, or other such device, may extract the embedded virtual objects from 

26 the video signal and provide the data to the modular connector 700 in the reception site 30. 

27 In another embodiment, virtual objects may be embedded within the audio signal, 

28 requiring an appropriate audio-capable modular connector 700 in the reception site 30 to extract 

29 the virtual obj ects from the audio signal. In one embodiment, the audio signal is an analog audio 

30 signal. In another embodiment, the audio signal is a digital audio signal. 
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1 In yet another embodiment, the signal is a spread spectrum signal containing a digital 

2 data stream, requiring an appropriate spread spectrum receiver modular connector 700 in the 

3 reception site 30 to extract the virtual objects. In this embodiment, the spread spectrum signal is 

4 transmitted in the same bandwidth as the video or audio signal, but below the noise level. 

5 Figure 27 presents embodiments associated with the delivery of virtual objects over a 

6 satellite-broadcast system 2901 to a reception site 30. Virtual objects are provided to the 

7 delivery network 1 1 by the object delivery center 15 or directly by the operations center 10. 

8 Alternatively, content 36 and virtual objects may be provided to the reception site 30 from the 

9 object delivery center 15 or from the local insertion center 20 using delivery network 12. The 

10 signal is delivered over the satellite broadcast system 2901 delivery network. The signal may 

11 provide for the delivery of virtual objects, content 36 containing virtual object locations, and 

12 reception site configuration and control information. The signal may also provide for virtual 

13 object viewing data and interactive virtual object requests from the reception site 30 to the local 

14 data collection center 40, to the central data collection center 50, or to the interactive object 

15 service center 60 using delivery network 14 or the signal may be a means to provide access to 

16 the Internet or other public network through which virtual objects or content 36 are delivered, 

1 7 The satellite broadcast system 2901 can be a direct broadcast system like DirecTV and EchoStar, 

18 a direct to home satellite broadcast system, video network distribution broadcast system, a 

19 point-to-point or point-to-multipoint data VS AT system, a digital audio broadcast system like 

20 WorldSpace, Sirius - formerly CD Radio, or XM, or a mobile data and telephony satellite 

21 broadcast system like Iridium, Teledesic, or Globalstar. Alternatively, the satellite broadcast 

22 system can be regionalized broadcast services or store and forward communication services 

23 hosted on high flying balloons or on airplanes that provide communication repeater services to 

24 an small geographic region. The signal over the satellite broadcast system may be generated by 

25 a satellite data modem, in which an external satellite data receiver 2902 is used to receive the 

26 signal and provide the embedded virtual obj ects to the reception site 30 modular connector 700 

27 for processing. Alternatively, the reception site 30 may contain an internal satellite receiver 

28 2905 , which receives the signal and provides the virtual obj ects to the modular connector 700 in 

29 the reception site 30 for processing. 

30 In another embodiment, the signal delivered over the satellite broadcast system is a 

31 video signal. Li one embodiment, the video signal is an analog video signal. In another 
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1 embodiment, the video signal is a digital video signal. The reception site 30 may contain an 

2 internal satellite video receiver 2906 to process the signal, and provide the embedded virtual 

3 objects to the modular connector 700. A satellite receiver 2903, or other device capable of 

4 receiving a satellite video signal, such as a TV, or PC with satellite receiver, may process the 

5 video signal and deliver the video signal to the modular connector 700 in the reception site 30, 

6 which extracts the embedded virtual objects. Alternately, the satellite receiver 2903, or other 

7 such device, may extract the embedded virtual obj ects from the video signal and provide the data 

8 to the modular connector in the reception site 258. 

9 In another embodiment, virtual objects may be embedded within the audio signal, 

10 requiring an appropriate audio-capable modular connector 700 in the reception site 30 to extract 

1 1 the virtual obj ects from the audio signal. In one embodiment, the audio signal is an analog audio 

12 signal. In another embodiment, the audio signal is a digital audio signal. 

13 In yet another embodiment, the signal is a spread spectrum signal containing a digital 

14 data stream, requiring an appropriate spread spectrum receiver modular connector 700 in the 

15 reception site 30 to extract the virtual objects. In this embodiment, the spread spectrum signal is 

16 transmitted in the same bandwidth as the video or audio signal, but below the noise level 

17 Figure 28 presents embodiments associated with the delivery of virtual objects over a 

18 wired data network 3001 to a reception site 30. Virtual objects are provided to the delivery 

19 network 11 by the object delivery center 15 or directly by the operations center 10. 

20 Alternatively, content 36 and virtual objects may be provided to the reception site 30 from the 

21 object delivery center 1 5 or from the local insertion center 20 using delivery network 12. The 

22 signal is delivered over the wired data network 300 1 delivery network. The signal may provide 

23 for the delivery of virtual objects, content 36 containing virtual object locations, and reception 

24 site configuration and control information. The signal may also provide for virtual object 

25 viewing data and interactive virtual object requests from the reception site 30 to the local data 

26 collection center 40, to the central data collection center 50, or to the interactive object service 

27 center 60 using delivery network 14 or the signal may be a means to provide access to the 

28 Internet or other public network through which virtual objects or content 36 are delivered. The 

29 wired data network 3001 can be metallic wire or fiber, supporting any of a number of 

30 communication standards including HDSL, ADSL, DSL, ISDN, Tl , T3, SONET, ATM, X.25, 

3 1 frame relay, Switched MultiMegabit Data Service (SMDS), or others. The signal sent over the 
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1 wired data network may be generated by a data modem or transmission device, in which the 

2 appropriate modem, interface device, or Data Terminating Equipment (DTE) device is used to 

3 receive the signal and provide the embedded virtual objects to the reception site 30 modular 

4 connector 700 for processing. Embodiments of such receiving devices are shown in Figure 28 as 

5 HDSL modem 3002, ADSL modem 3003, DSL modem 3003, ISDN Terminal equipment (TE) 

6 device 3005, Tl Digital service unit (DSU) 3006, T3 DSU 3007, Fiber user network interface 

7 device (UNI) 3008, ATM UNI 3009, X.25 DTE 3010, Frame relay assembler/disassembler 

8 (FRAD) 301 1, and SMDS subscriber network interface device (SNI) 3012. Alternatively, the 

9 reception site 30 may contain an internal modem or DTE 3013, which receives one or more 

10 signal types and provides the received signal with embedded virtual objects to the modular 

11 connector 700 in the reception site 30 for processing. Finally, the reception site 30 may be 

12 attached to a wired LAN using a transceiver. In this embodiment, virtual objects may be 

13 delivered over the LAN at any time. 

14 Figure 29 presents embodiments associated with the delivery of virtual objects using the 

15 public switched telephony network (PSTN) 3101 to a reception site 30. Virtual objects are 

1 6 provided to the delivery network 1 1 by the obj ect delivery center 1 5 or directly by the operations 

17 center 10. Alternatively, content 36 and virtual objects may be provided to the reception site 30 

18 from the object delivery center 15 or from the local insertion center 20 using delivery network 

19 12. The signal is delivered over the PSTN 3101 delivery network. The signal may provide for 

20 the delivery of virtual objects, content 36 containing virtual object locations, and reception site 

2 1 configuration and control information. The signal may also provide for virtual object viewing 

22 data and interactive virtual object requests from the reception site 30 to the local data collection 

23 center 40, to the central data collection center 50, or to the interactive object service center 60 

24 using delivery network 1 4 or the signal may be a means to provide access to the Internet or other 

25 public network through which virtual objects or content 36 are delivered. The signal sent over 

26 the PSTN may be generated by a data modem or transmission device, in which the appropriate 

27 modem 3102 is used to receive the signal and provide the embedded virtual objects to the 

28 modular connector 700 in the reception site 30 for processing. Alternatively, the reception site 

29 30 may contain an internal modem 3103, which receives the signal and provides the received 

30 signal with embedded virtual objects to the modular connector 700 in the reception site 30 for 

31 processing, 
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1 Figure 30 presents embodiments associated with the delivery of virtual objects using 

2 wireless personal communications system (PCS) 320 1 to a reception site 30. Virtual objects are 

3 provided to the delivery network 1 1 by the obj ect delivery center 1 5 or directly by the operations 

4 center 10. Alternatively, content 36 and virtual objects may be provided to the reception site 30 

5 from the object delivery center 15 or from the local insertion center 20 using delivery network 

6 12. The signal is then delivered over the PCS network 3201 delivery network. The wireless 

7 PCS system may be, for example a wireless LAN, digital cellular telephony network, analog 

8 cellular telephony network, digital cellular radio system, analog cellular radio system, digital 

9 pager network, analog pager network, or Personal Communication Network (PCN). The signal 

10 may provide for the delivery of virtual objects, content 36 containing virtual object locations, 

1 1 and reception site configuration and control information. The signal may also provide for virtual 

12 object viewing data and interactive virtual object requests from the reception site 30 to the local 

13 data collection center 40, to the central data collection center 50, or to the interactive object 

14 service center 60 using delivery network 14 or the signal may be a means to provide access to 

1 5 the Internet or other public network through which virtual objects or content 36 are delivered. A 

16 wireless PCS receiver 3202 is used to receive the signal and provide the embedded virtual 

17 objects to the modular connector 700 in the reception site 30 for processing. Alternatively, the 

1 8 reception site 25 8 may contain an internal wireless PCS receiver 3203, which receives the signal 

1 9 and provides the received signal with embedded virtual obj ects to the modular connector 700 in 

20 the reception site 30 for processing. 

21 Figure 3 1 depicts several embodiments associated with the delivery of virtual objects 

22 using a national or local television broadcaster's signal. Virtual obj ects are provided to the either 

23 the national broadcaster 1110, the broadcast affiliate 1 1 12, or the local cable system 1 1 14 by the 

24 object delivery center 15 or directly by the operations center 10. The signal from the national 

25 broadcaster 1110 can be delivered to reception site 30', 30" or 30"' using a satellite system 1 122, 

26 using a broadcast affiliate 1112 terrestrially, or using a local cable system 1114. Alternatively, 

27 the local television broadcast affiliate 1112 can originate the signal which can be delivered to the 

28 reception site 30', 30" or 30'" terrestrially, or using a local cable system 1 1 14. The signal may 

29 provide for the delivery of virtual objects, content 36 containing virtual object locations, and 

30 reception site configuration and control information. The signal may also provide for virtual 

3 1 object viewing data and interactive virtual object requests from the reception sites 30', 30", and 

60 5221 



1 30'" to the local data collection center 40, to the central data collection center 50, or to the 

2 interactive object service center 60 using delivery network 14 or the signal may be a means to 

3 provide access to the Internet or other public network through which virtual obj ects or content 36 

4 are delivered. In one embodiment, the video signal is an analog video signal and the virtual 

5 objects is embedded in the video signal. In another embodiment, the video signal is a digital 

6 video signal and the virtual objects are carried as an independent data stream. In another 

7 embodiment, virtual obj ects may be embedded within the audio signal. In one embodiment, the 

8 audio signal is an analog audio signal. In another embodiment, the audio signal is a digital audio 

9 signal. 

10 In yet another embodiment, the signal is a spread spectrum signal containing a digital 

1 1 data stream, requiring an appropriate spread spectrum receiver modular connector, such as the 

12 connector 700 of Figure 33, in the reception site 30*, 30" or 30 m to extract the virtual objects. In 

1 3 this embodiment, the spread spectrum signal is transmitted in the same bandwidth as the video or 

14 audio signal, but below the noise level. 

15 Alternatively, several embodiments are associated with the delivery of virtual objects 

16 using a national or local radio broadcaster's signal. The signal from the national radio 

17 broadcaster can be delivered to the reception site 30', 30" or 30"' using the satellite system 1 122, 

1 8 or using a broadcast affiliate 1 1 22. Alternatively, the radio broadcast affiliate 1 1 22 can originate 

19 the signal, which can be delivered to the reception site 30', 30" or 30'", terrestrially. In one 

20 embodiment, the audio signal is an analog audio signal and the virtual objects is embedded in the 

21 audio signal. In another embodiment, the audio signal is a digital audio signal and the virtual 

22 objects are carried as an independent data stream. In yet another embodiment, the virtual objects 

23 are embedded in a sub-carrier of the analog audio broadcast. In another embodiment, the signal 

24 is a spread spectrum signal containing a digital data stream, requiring an appropriate spread 

25 spectrum receiver modular connector 700 in the reception site 30', 30" or 30'" to extract the 

26 virtual objects. In this embodiment, the spread spectrum signal is transmitted in the same 

27 bandwidth as the audio signal, but below the noise level. 

28 A local insertion center 20 or multiple local insertion centers may optionally be used to 

29 insert virtual objects into content 36 provided by an operations center 10 or another local 

30 insertion center 20, and any other content source. A local insertion center 20 may perform the 

31 same functions as an operations center 10. Figure 32 depicts a local insertion center 20. As 
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1 shown in Figure 32, the local insertion center 20 includes a virtual object location definer 100', a 

2 virtual object selector 200', and a targeted virtual object management system 300 ! (TVOMS) 

3 which are identical to the virtual object location definer 100, a virtual object selector 200, and a 

4 targeted virtual object management system 300 (TVOMS) of an operations center 10. A local 

5 insertion center 20 may detect existing virtual obj ect locations in content 36 and replace existing 

6 virtual objects with new virtual objects, delete existing virtual objects, or add new virtual objects 

7 in existing virtual object locations and target the virtual objects to reception sites or groups of 

8 reception sites. Alternatively, a local insertion center 20 may create new virtual object locations 

9 and insert and target virtual obj ects within these new virtual obj ect locations using the processes 

10 defined for the operations center 10. The local insertion center 20 may modify an existing or 

1 1 generate a new retrieval plan or generate a new or modify an existing group assignment matrix 

1 2 for distribution to reception sites . 

13 Figure 33 depicts an example of a reception site 30 in more detail The modular 

14 connector 700 may handle all interactions with a reception site 30. Programming content 36 

1 5 with virtual obj ect locations and metadata packets containing placement guidelines, mattes, and 

1 6 retrieval plans are received by the reception site modular connector 700 and passed to the virtual 

17 object extractor processor 780. The virtual object extractor processor 780 removes any virtual 

1 8 objects from the received signal and the retrieval plan information and routes the virtual objects 

1 9 and retrieval plan to the storage management processor 710. The storage management processor 

20 7 1 0 uses the retrieval plan to determine which virtual obj ects are destined to the reception site 3 0 

21 and saves the required virtual objects in virtual object storage 720. In an alternative 

22 embodiment, virtual objects may be received by the reception site 30 independent of the 

23 programming content 36, 

24 The programming content 36 with virtual object locations is then passed to the virtual 

25 object location detector processor 750. Information received about virtual object locations is 

26 extracted from the programming content 36 and passed to the selector processor 740 which 

27 coordinates with the storage management processor 710 to determine the appropriate virtual 

28 object 38 to place into each virtual object location 37 based on placement guidelines and 

29 available virtual objects stored in the virtual object storage 720. The storage management 

30 processor 710 retrieves the appropriate virtual object 3 8 for one or more virtual object locations 
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1 contained in the content 36 from the virtual object storage 720. Virtual objects are passed from 

2 the storage management processor 710 to the virtual object insertion processor 760, 

3 Programming content 36 with virtual object locations is passed from the virtual object 

4 location detector processor 750 to the content buffer 790 where the programming content 36 is 

5 stored for a fixed period of time and then played out of the content buffer 790 to the virtual 

6 obj ect insertion processor 760. If a virtual obj ect 3 8 is available for placement in a virtual obj ect 

7 location 3 7, the virtual obj ect 3 8 is inserted into the appropriate virtual obj ect location 3 7 by the 

8 virtual object insertion processor 760. 

9 In one embodiment, the virtual obj ect location 37 may require that an embedded virtual 

10 object 38 be placed within the content 36. The virtual object insertion processor 760 may use 

1 1 techniques for the insertion of embedded virtual objects which are described in detail in U.S. 

12 Patents 5,953,076, to Astle, Brian; and Das, Subhodev; titled System and Method of Real Time 

13 Insertions into Video Using Adaptive Occlusion with a Synthetic Reference Image; 5,892,554, to 

14 DiCicco, Darrell; and Fant, Karl; entitled System and Method for Inserting Static and Dynamic 

1 5 Images into a Live Video Broadcast; 5,5 1 5,485, to Luquet, Andre; and Rebuffet, Michel; entitled 

1 6 Method and Device for Modifying a Zone in Successive Images; 5,903,3 1 7, to Sharir, Avi; and 

17 Tamir, Michael; entitled Apparatus and Method for Detecting, Identifying and Incorporation 

18 Advertisements in a Video; and the MPEG4 standard, the disclosure of which are hereby 

1 9 incorporated by reference. 

20 In another embodiment, when the virtual obj ect location 3 7 may require that an overlaid 

2 1 virtual obj ect 3 8 be placed within the content 3 6 . The virtual obj ect insertion processor 760 may 

22 use techniques for the overlaying of virtual obj ects which are described in detail in U. S . Patents 

23 4,319,266 to Bannister, Richard S.; entitled Chroma Keying System; 4,999,709 to Yamazaki, 

24 Hiroshi; and Okazaki, Sakae; entitled Apparatus for Inserting Title Pictures; 5,249,039, to 

25 Chaplin, Daniel J,; entitled Chroma Key Method and Apparatus; and 5,233,423 to Jernigan, 

26 Forest E>; and Bingham, Joseph; entitled Embedded Commercials within a Television Receiver 

27 using an Integrated Electronic Billboard, the disclosure of which are hereby incorporated by 

28 reference. Programming content 36 with embedded and overlaid virtual objects is passed to an 

29 optional interactive object processor 770. 

30 Preferably, when a virtual object 38 is placed into a virtual object location 37, the 

31 selector processor 740 records the event in the placement log 730. The placement log 730 
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1 provides viewing data to the local data collection center 40 or the central data collection center 

2 50, where the information can be used for future virtual object targeting or billing of virtual 

3 obj ect providers, for example, advertisers. The selector processor 740 can be provided targeting 

4 algorithm updates from external sources. 

5 A local data collection center 40 is depicted in Figure 34. The local data collection 

6 center 40 collects, processes, and stores data from reception sites, from a central data collection 

7 center 50, or other sources. The data collected about reception sites may be provided to a local 

8 insertion center 20 to be used in targeting virtual objects in content 36. Alternatively, the data 

9 collected from receptions site may be provided to a central data collection center 50 to be used in 

10 targeting virtual objects in content 36 by an operations center 10. As shown in Figure 34, 

1 1 communications to and from the local data collection center 40 over a delivery network may be 

12 done using modular connector 700. An interface 41 receives information from reception sites. 

1 3 The interface 4 1 can include a workstation, such as the workstation 44, for example, from which 

14 an operator manually enters reception site information. Alternately, reception site information 

15 can be automatically entered at the interface 41 by downloading from an off-site database, the 

16 Internet, a storage medium, such as a CD-ROM or a floppy disk, and by collecting the 

17 information directly from the individual reception sites using modular connector 700, A 

1 8 processor 42 processes the received reception site information and organizes the information for 

19 use and stores information in database 43. 

20 A central data collection center 50 is depicted in Figure 35. The central data collection 

21 center 50 collects, processes, and stores data from reception sites, from local data collection 

22 centers, or other sources. The data collected about reception sites may be provided to a local 

23 insertion center 20 or local data collection center 40 to be used in targeting virtual objects in 

24 content 36. Alternatively, the data collected from reception site may be provided to an 

25 operations center 1 0 to be used in targeting virtual obj ects in content 36. As shown in Figure 34, 

26 communications to and from the central data collection center 50 over a delivery network may be 

27 done using modular connector 700. An interface 5 1 receives information about reception sites. 

28 The interface 5 1 can include a workstation, such as the workstation 54, for example, from which 

29 an operator manually enters reception site information. Alternately, reception site information 

30 can be automatically entered at the interface 51 by downloading from an off-site database, the 

31 Internet, a storage medium, such as a CD-ROM or a floppy disk, and by collecting the 

64 5221 



1 information directly from the individual reception sites using modular connector 700. A 

2 processor 52 processes the received reception site information and organizes the information for 

3 use and stores information in database 53. 

4 Returning to Figure 33, an external trigger may be received by the interactive object 

5 processor 770 indicating the subscriber has selected an interactive virtual object 38. 

6 Alternatively, the interactive object processor 770 may be capable of being configured to 

7 automatically process all interactive virtual objects received. Figure 36 depicts the steps the 

8 interactive object processor 770 performs upon receipt of an external trigger related to an 

9 interactive virtual object 38. The process begins with the start ellipse 550. The interactive 

10 object processor 770 receives the trigger as shown in block 551. The interactive object processor 

11 770 then retrieves the interactive virtual object trigger action 56 from the interactive virtual 

12 object 38, as shown in block 552. The interactive object processor 770 determines if the 

1 3 interactive virtual obj ect trigger action 5 6 requires initiation of an interactive request to a remote 

14 site, as shown in block 553. As shown in block 554, if only local action at the reception site 30 

15 is required, the interactive object processor 770 initiates any local processing required by the 

16 interactive virtual object trigger action 56 associated with the interactive virtual object 38. After 

17 initiation of any actions required by the interactive virtual object trigger action 56, the process 

18 ends with ellipse 559. If, in block 553, the interactive object processor 770 determines that the 

1 9 interactive virtual obj ect trigger action 5 6 requires initiation of an interactive request to a remote 

20 site, the interactive object processor 770 initiates the sending of the interactive request with the 

21 virtual object identifier 58, as shown in block 555. The interactive object processor 770 passes 

22 the interactive request to the modular connector 700. The modular connector 700, in turn, passes 

23 the interactive request to the interactive object servicing center 60. The interactive object 

24 processor 770 awaits for any interactive response, as shown in block 556. The interactive object 

25 servicing center 60 may process the interactive request and may respond back to the reception 

26 site 30 with an interactive response. The interactive object processor 770 receives and processes 

27 any interactive response received from the interactive object servicing center 60, as shown in 

28 block 557. The interactive obj ect processor 770 then initiates any further actions required by the 

29 interactive response or the initial interactive virtual obj ect trigger action 56 received, as shown in 

30 block 558. The process ends with ellipse 559. 
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1 A diagrammatic representation of an interactive virtual obj ect 3 8 is presented in Figure 

2 37. Each interactive virtual object 38 is identified by a unique virtual object identifier 58, This 

3 virtual object identifier 58 may be assigned by the virtual object management center 55, and 

4 provided to the interactive object servicing center 60, with the interactive virtual object response 

5 management guidelines 57 associated with the interactive virtual object 38. Virtual object 

6 placement rules and guidelines 151 may be delivered with the virtual object 38 to provide 

7 guidance to the reception site 30 in managing the insertion of virtual objects into content 36. 

8 The virtual object digital module 59 is the actual digital representation of the virtual object 38 

9 that is created and stored at the operations center 10 and stored by the storage management 

10 processor 710 at the reception site 30 for use by the virtual object insertion processor 760 in 

1 1 recreating a visual representation of the virtual object 38. The interactive virtual object trigger 
*% 12 action 56 associated with an interactive virtual object 38 provides a definition of the action 

13 required to be taken by the interactive object processor 770 upon subscriber selection of an 

if: 14 interactive virtual object 38. The interactive virtual object trigger action 56 may result in the 

f ; 1 5 initiation of processing by the interactive obj ect processor 770. Alternatively, or in addition, the 

i t 16 interactive virtual object trigger action 56 may result in the initiation of an interactive request to 

17 an interactive virtual object servicing center 60. Alternatively, or in addition, the interactive 

y 18 virtual object trigger action 56 may result in the interactive object processor 770 providing 

iT 1 9 interactive virtual obj ect control 1 5 3 commands or providing an optional virtual obj ect software 

20 applet 125 to an device external to the reception site 30 for additional processing therein. The 

21 external device may consist of a television set-top terminal, computer with Internet access, 

22 digital television receiver, or other device capable of processing the interactive virtual object 

23 control 153 commands or optional virtual object software applet 1 52. The optional virtual object 

24 software applet 1 52 provides software that may be initiated by the interactive object processor 

25 770 or provided to an external device to be initiated. A difference between an interactive virtual 

26 object 38 and a virtual object 38 that is not interactive is the placement of information in the 

27 interactive virtual object trigger action 56 field of a virtual object 38 and the placement of 

28 information in the optional virtual object software applet 152 field of a virtual object 38. 

29 An interactive virtual object management center 55 is depicted in Figure 38. The 

30 interactive virtual object management center 55 generates interactive virtual objects and provides 

31 them to the operations center 10 or any other location where interactive object insertion or 
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1 delivery may take place. The interactive virtual object management center 55 provides 

2 interactive virtual object response management guidelines 57 to an interactive object servicing 

3 center 60 which may be used by the interactive object servicing center 60 to determine the 

4 appropriate response upon receipt of an interactive request from a reception site 30. As shown in 

5 Figure 38, communications to the interactive object servicing center 60 and to the operations 

6 center 10, or to any other location placing virtual objects into content 36, may be done using 

7 modular connector 700. An interface 161 provides interactive virtual objects to the operations 

8 center 1 0 and provides the interactive virtual object response management guidelines 57 to an 

9 interactive object servicing center 60. The interface 161 can include a workstation, such as the 

10 workstation 164, for example, from which an operator manually enters interactive virtual object 

1 1 definition information used to create the interactive virtual obj ect 3 8 . A processor 1 62 processes 

12 the interactive virtual object definition, performs the appropriate action, and stores interactive 

13 virtual object information in database 163. 

14 Interactive virtual objects may be used for a variety applications resulting in the 

15 initiation of processing at the reception site 30 or initiation of processing by an external device 

16 accessible by the reception site 30. In one embodiment, selection of an interactive virtual object 

17 38 may result in the interactive object processor 770 retrieving an optional virtual object 

18 software applet 1 52 from the interactive virtual object 38 and initiating the optional virtual object 

1 9 software applet 1 52 at the interactive obj ect processor 770, or storing the optional virtual obj ect 

20 software applet 152 in an interactive object processor storage 154 for future initiation at the 

21 reception site 30, or providing the optional virtual object software applet 152 to an external 

22 device. In another embodiment, selection of an interactive virtual object 38 may result in the 

23 interactive obj ect processor 770 initiating an optional virtual object software applet 1 52 that may 

24 have been previously received and stored in an interactive object processor storage 154 or 

25 resident elsewhere at the reception site 30. In yet another embodiment, selection of an 

26 interactive virtual object 38 may result in the interactive object processor 770 generating an 

27 interactive virtual object control command 153 to be provided to a device external to the 

28 reception site 30. In one embodiment, the interactive virtual object control command 153 may 

29 notify the external device to select a different language of audio to be associated with the content 

30 36. In another embodiment, the interactive virtual object control command 153 may notify the 

3 1 external device to initiate the printing of a coupon or document associated with the interactive 
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1 virtual obj ect 3 8 . In yet another embodiment, the interactive virtual obj ect control command 1 53 

2 may notify the external device to cause the selection of a different source for content 36, a 

3 different channel of content, or different camera angle for the video content 36 being viewed. 

4 Alternatively, in one embodiment, selection of an interactive virtual obj ect 3 8 may result 

5 in the interactive object processor 770 providing interactive virtual object selection data 155 to 

6 the selector processor 740 to control which virtual objects are to be displayed at the reception 

7 site 30. In this embodiment, the selector processor 740 may control which virtual object 38 is 

8 placed in a virtual object location 37 based on the interactive virtual object 38 being selected or 

9 past interactive virtual obj ects selected. Virtual obj ects associated with a virtual obj ect location 

10 37 may have different fonts or font sizes, allowing the subscriber to zoom in or zoom out from 

11 textual information displayed as a virtual object 38 by selecting the appropriate interactive 

12 virtual object 38. Virtual objects associated with a virtual object location 37 may have different 

13 orientations, allowing the subscriber to select the desired orientation to display by selecting the 

14 appropriate interactive virtual object 38. Virtual objects associated with a virtual object location 

15 37 may have multiple layers, allowing the subscriber to peel away layers one by one by selecting 

16 the appropriate interactive virtual object 38. Virtual objects associated with a virtual object 

17 location 37 may be opaque in the nature, allowing the subscriber to select whether to make the 

18 virtual object 38 transparent, displaying the underlying image by selecting the appropriate 

1 9 interactive virtual obj ect 3 8 . Selection of such an interactive virtual obj ect 3 8 may be password 

20 protected, to allow, for example, a parental control feature, where an opaque virtual object 38 is 

2 1 not removable, exposing the content underneath, unless the appropriate password is entered by 

22 the subscriber when selecting the interactive virtual object 38, 

23 In another embodiment, selection of an interactive virtual object 38 may result in the 

24 interactive object processor 770 providing interactive virtual object selection data 155 to the 

25 placement log 730 to record which interactive virtual objects have been viewed or selected by a 

26 subscriber. The viewing information may then be provided to the local data collection center 40 

27 or the central data collection center 50 to be used for future virtual object targeting purposes, 

28 In yet another embodiment, selection of an interactive virtual object 3 8 may result in the 

29 interactive obj ect processor 770 providing placement control 1 5 6 to the virtual obj ect insertion 

30 processor 760, affecting the location of placement of a virtual object 38 in content 36. 
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1 In another embodiment, selection of an interactive virtual object 38 may result in the 

2 interactive object processor 770 accessing an Internet website and displaying a Webpage on 

3 display 35 at the reception site 30 or on an external device. 

4 An interactive object servicing center 60 is depicted in Figure 39. The interactive object 

5 servicing center 60 processes interactive requests and formulates responses to such requests. 

6 Figure 40 presents the process the interactive object servicing center 60 performs. The process 

7 begins with block 4500. In block 450 1, the interactive object servicing center 60 receives 

8 interactive requests from reception sites. In block 4502, the interactive object servicing center 

9 60 determines the appropriate action to be performed based on the received interactive request. 

10 In block 4503, the interactive object servicing center 60 performs the appropriate action based on 

11 the received interactive request and the interactive virtual object response management 

12 guidelines 57 previously provided by the interactive virtual object management center 55. In 

1 3 block 4504, the interactive object servicing center 60 replies to the requesting reception site with 

14 an interactive response. Interactive virtual objects may be used for a variety applications 

15 resulting in the generation of an interactive request. In one embodiment, an interactive virtual 

16 object 38 may result in the generation of an interactive request whereby the interactive object 

17 servicing center 60 logs that an interactive virtual object was selected by a reception site 30. 

18 This logged information may be used in refining the virtual object targeting algorithm, as this 

19 logged information provides a positive indication that a particular segment of content was 

20 viewed by a subscriber at the reception site 30. This logged information may alternatively be 

21 used by the content provider to bill an advertiser, as the interactive request serves as a positive 

22 indication that an advertisement was actively viewed by a subscriber and solicited an action on 

23 the part of the subscriber. In another embodiment, an interactive virtual object 38 may result in 

24 the generation of an interactive request whereby the interactive object servicing center 60 

25 initiates an electronic transaction that is associated with the selected interactive virtual obj ect 3 8 . 

26 For example, the subscriber may have selected an interactive virtual object 38 in a video version 

27 of a product catalog and the selection of that interactive virtual object 38 initiates the purchase of 

28 the product associated with the selected interactive virtual object 38. In yet another embodiment, 

29 selection of an interactive virtual object 38, for example a short video clip on an electronic 

30 program guide that is an interactive virtual object, may result in the generation of an interactive 

3 1 request for a video on demand or pay per view purchase, whereby the interactive object servicing 
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1 center 60 processes the interactive request from the reception site 30 and notifies a video server 

2 to begin playback of video on demand content to the requesting reception site 30. In another 

3 embodiment, an interactive virtual object 38 may result in the generation of an interactive 

4 request whereby the interactive object servicing center 60 responds to the interactive request 

5 with an interactive response that contains a software applet to be run on a processor at the 

6 reception site 30. In yet another embodiment, an interactive virtual object 38 may result in the 

7 generation of an interactive request whereby the interactive obj ect servicing center 60 responds 

8 to the interactive request with a webpage to be displayed at the reception site 30. 

9 As shown in Figure 39, communications to and from the interactive object servicing 

10 center 60 over a delivery network may be done using modular connector 700. An interface 61 

11 receives interactive requests from reception sites and receives the interactive virtual object 

12 response management guidelines 57 from the interactive virtual object management center 55. 

1 3 The interface 6 1 can include a workstation, such as the workstation 64, for example, from which 

14 an operator manually enters interactive request behavior for the interactive object servicing 

15 center 60 or can modify the interactive virtual object response management guidelines 57 

1 6 received from the interactive virtual obj ect management center 5 5 . A processor 62 processes the 

17 received interactive requests and received interactive virtual object response management 

18 guidelines 57, performs the appropriate action, retrieving information from database 63 to 

19 perform the actions and storing transaction information in database 63 to record the transaction 

20 event. 

21 A variety of interactive virtual object targeting delivery systems have been described. 

22 One of ordinary skill in the art will recognize that the above description is that of preferred 

23 embodiments of the invention and the various changes and modification may be made thereto 

24 without departing from the spirit and scope of the invention as defined in the following claims. 
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